Moving from 16 bay to 42 hot swap bays --- case check?


DieFalse

Recommended Posts

So,   I just did a thing..... A thing I may love or hate myself for.

 

I currently have a Norco 16 bay case that I loved at the time and now hate due to how cheaply it is made.  While it has served me well for a year, its time to move on.

 

I found a 8U-42bay case that has great design qualities, but little following.   So it may be junk or it may be gold.

 

I wanted to be able to use ALL of my PRO license capable drives as currently 16x4tb is not enough.  When I get close to filling up 28 data drives with 2 parity,  I will just add a second server.  But for now, I want my cache drives to be expandable, unassigned available for VMs and well a couple of dockers I want on independent drives.

 

What better way than 42 bays?  And I can add my 5.25 to 6 2.5" ssd into the 5.25 bay.   I am posting here because I will be moving my current threadripper into it, and wonder if any one here uses or used istarusa cases?

 

Product Link

http://www.istarusa.com/en/istarusa/products.php?model=E8M42 

Link to comment

Well, I found out why I was able to discover why the case was available at an amazing price of 350.00.  there is no power supply.   It takes a special 1U power supply and it's 1000.00+ by itself.  I got lucky yet again and Newegg had a reconditioned one for 275.00. So overall still a great price.   Until it comes in I am at a stand still.   The construction of this case is amazing though.  It's like a tank and decently heavy like most supermicros I've dealt with.  Rolled steel.   I imagine it will weight as much as a tank when done also. So I checked and rails are not recommended due to weight, they recommend a heavy duty sliding tray.  I wonder how that will work with my rolling 18u chassis.

Link to comment

The PSU part really headache, so special, all redundancy type PSU price so crazy. I never think own such tank. ( USD350 for new ? )

How you connect 48drive ? By native 8087 / 8643 cable, it need 10+ ... I would like go to expander blackplane solution more.

 

For me if really like that tank, I won't spend 1000 for PSU, so the only option will be reconditioned one or DIY ( may be lost some bays ).

 

USD350 for 8U case was good deal.

Edited by Benson
Link to comment

The case was business purchased and then returned.  I got it as a "returned - new" condition.   It looks new and only missing the PSU which I lucked out and got with the newegg deal for 275.   so USD 353+275    $628 total for the case.  When the PSU comes in I will know more and post pictures as well.   Its SFF8087 mini sas backplaned with built in expander.  I dont yet have a plan for enabling all 42 bays but will see.  

 

My biggest desire is enabling the following:

2 lower bays = parity drives

28 data drives for the max allowed on Pro

4 drives for individual VMs

2 drives for individual dockers (SQL dbs and Nextcloud)

6 (ssds in a 5.25 to 2.5 converter) for cache.

Total functioning drives: 42

 

But will start with:

18 Data drives

2 Parity

6 SSDs in the 5.25 converter

Total: 26

 

Anyone with insight on the best way to enable all 42 drives?

Link to comment
4 minutes ago, fmp4m said:

Wondering how to tell if that 40 port would work with unraid or not.

Work, yes, recommend no, besides being a Marvell chipset, same as the SAS2LP, and a known source of issues, it also uses 4 SATA ports multipliers, another no-no, also performance won't be good.

Link to comment

If the budget allows it and assuming you have another PCIe slot, get a SAS3 LSI like the 9300-8i plus a SAS3 expander like the Intel RES3TV360, that together with your existing LSI will give you enough ports with lots of bandwidth.

 

If that's too expensive get another SAS2 HBA like the 9207-8i and an Intel SAS2 expander like the RES2CV360, a little less total bandwidth but miles better than the RocketRAID, both in terms of speed and more importantly much more reliable.

Link to comment

Right now I have "LSI SAS 9201-16i".  The new case only has "SAS I/II & SATA I/II/III" according to specs.  So with that, do you still recommend the SAS3?  If so,  Would it be best to go with 2 of those and the expander?

 

Either solution, how exactly does one properly use those expanders,  I have never had one?

 

Link to comment
5 hours ago, fmp4m said:

I need another 9201-16i and a RES2CV360?

An 8 port HBA would be enough, preferably a PCIe 3.0 one, like 9207-8i, both miniSAS ports connect to the expander and the expander connects to the backplane, up to 28 devices total, together with the existing 9201-16i you can connect up to 44 devices.

Link to comment

That makes sense however,  wouldnt the expander card be sharing the bandwidth of the 8 sas ports into the 28ports?  If thats the case, I would imagine to get equal bandwidth across the case, I would be better with 2x8i with both being connected to expander cards.  Unless there is some sort of trickery I am not aware of that would allow the drives to all talk at the same speeds otherwise?

 

You know way more than I do,  so I am looking for advice, This will be a long term build so I have the budget for it, just want to make sure I get the right stuff so I can upgrade from 4tb to 8-10tb later each bay.  Having the Threadripper keeps it in my line of use life for quite a while, so I want the same with the 40 sas connected bays.

 

Since the SAS3 cards will provide more bandwidth and work with the case,  I am sourcing those now (posted in WTB here first).  Just wanna make sure I am grabbing the right quantities for best overall design.

Link to comment
1 hour ago, fmp4m said:

wouldnt the expander card be sharing the bandwidth of the 8 sas ports into the 28ports?

Yes, but with a SAS2 PCIe 3.0 HBA it still has 4400MB/s usable total, so around 157MB/ per disks if fully loaded, a SAS3 HBA + expander would give you around 6000MB/s total.

 

1 hour ago, fmp4m said:

I would be better with 2x8i with both being connected to expander cards. 

Yes, if you can afford it, just divided 4400MB/s (if using the same PCIe 3.0 SAS2 HBAs) by the number of disks connected to see how much more.

 

1 hour ago, fmp4m said:

Since the SAS3 cards will provide more bandwidth

Yes, but only if used with a SAS3 expander from LSI.

 

Edit: See here for some real worlds tests:

 

Edited by johnnie.black
  • Upvote 1
Link to comment

Alright I have ordered 2x9300-8i's and 2xRES2CV360.  This should equal the bandwidth and be my best overall.   I really appreciate the follow up and answers to questions you have provided.

 

The PSU came in and is installed,  now waiting the cards and expanders to begin moving to the new case.  I have not really worked with SAS to SAS before and the connectors on the backplanes does not look familiar to me.  

 

With the 9300-8i's and expanders and the backplanes pictured above,  what cables will I need?  I could wait until the cards come in, but want to get started on moving it when they do with minimal downtime.

 

 

8i-cards.jpg

RES3TV360.jpeg

Link to comment

I have hit a small wall with the bottom two hot-swap bays.  The mfg says "Two bottom drive bays can be dedicated for OS Redundancy" which I initially took as "independent sas/sata drives"..... Turns out they are, but with 2 Sata Ports per Bay.  1 Blue 1 Black.  I have never seen this before,  I have reached out to istarusa and they have ZERO documentation on them and the tech sheet is useless.

 

Anyone have any experience with them?  Google is showing nothing about this specific backplane.  Some other searches suggest its "dual port means that the hdd/ssd has additional pins. These pins are connected to a second port which can be connected to another hba/raid controller for high avalibility."   So seeming I would only need to use the one port,  and if wanted HA/Redundancy (I dont think unraid would like it) connect the second to a different controller than the first.

 

I was able to snap some pics:

IMG_20190221_180834.thumb.jpg.17a19e4b634778b6fd88d8ecf9e57faa.jpgIMG_20190221_180648.thumb.jpg.fef8b64a0d58e0593294c6f0db1861ab.jpgIMG_20190221_180715.thumb.jpg.20febae6172b5bc0549e9801e5c9e554.jpg

Edited by fmp4m
Link to comment

Cables and 2x 8i cards came in.   Now waiting on expanders.   I hope I wired the two bottom bays correctly, I used sff8643 to sata break outs and put p1 and p2 of what will be expander 1 on the left, and p1 p2 of what will be expander 2 on the right.

 

That will make 2 ports per parity sas trays and then 5x4 rows of hdds per expander.   21 drives per expander. 22 of 24 ports used per expander and two uplinks to the 8i card each.

 

So many wires to cable manage. :(

 

 

Link to comment
  • 2 weeks later...

Alright,  got everything into the Behemoth this past weekend,  Started Parity Check and waiting for it to finish,  its slow for some reason,  but I will need for it to finish before checking on what the issue is.   Data RW tests before booting unraid is over 2GB/s for all drives RW at once which was about .9Gb/s faster than before.

 

A few questions:

 

1. I need to upgrade my last 2TB drive to a 8TB at the same time I need to add 2 8TB drives and add a 2nd 8TB Parity.  Can I do all of this at once since only one drive will be being rebuilt and the rest are new?  

 

2. Is there a way to speed up parity?  or Troubleshoot what is taking so long?   When I have Zero Dockers running the parity is around 200-300Mb/s for all drives at one time.   I have all new controllers, cords, and expanders now as seen above.

 

 

Link to comment
4 minutes ago, fmp4m said:

Can I do all of this at once since only one drive will be being rebuilt and the rest are new?  

No, not possible to rebuild and add drives at the same time, you can rebuild and and parity2 and when done add the remaining drives, or vice-versa.

 

5 minutes ago, fmp4m said:

Is there a way to speed up parity?  or Troubleshoot what is taking so long?

Grab diags during current check.

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.