ATLAS My Virtualized unRAID server


Recommended Posts

i just ordered some stuff to somewhat copy the ATLAS server

i ordered

SUPERMICRO MBD-X9SCM-F-O

XFX ATI Radeon HD 5670 1GB DDR3 VGA/DVI/HDMI PCI-Express Video Card HD567XZNL3

 

im shooting on getting the video to Hardware Passthough for my camera system ive seen sites saying that they can pass that video card

 

Err one major flaw. the SCM only has PCIe 8x/4X slots

the video card is PCIe 2.1 16x

 

you would probably be better off with an Intel desktop board (or similar?) for this  setup.

 

Your right... i looked this morning to see when my stuff was coming in and noticed the motherboard didn't have a x16 slot

 

i thought about a desktop board but i wanted IPMI since it will somewhat be headless

i have a SuperMicro Atom D510 and boy that IPMI is so nice :)

ill have to order a x1 slot video card and hope that works with passthough

 

If the back of the X1 connector on the motherboard is open (some are), you can use a x16 card in a X1 slot.

Link to comment

i just ordered some stuff to somewhat copy the ATLAS server

i ordered

SUPERMICRO MBD-X9SCM-F-O

XFX ATI Radeon HD 5670 1GB DDR3 VGA/DVI/HDMI PCI-Express Video Card HD567XZNL3

 

im shooting on getting the video to Hardware Passthough for my camera system ive seen sites saying that they can pass that video card

 

Err one major flaw. the SCM only has PCIe 8x/4X slots

the video card is PCIe 2.1 16x

 

you would probably be better off with an Intel desktop board (or similar?) for this  setup.

 

Your right... i looked this morning to see when my stuff was coming in and noticed the motherboard didn't have a x16 slot

 

i thought about a desktop board but i wanted IPMI since it will somewhat be headless

i have a SuperMicro Atom D510 and boy that IPMI is so nice :)

ill have to order a x1 slot video card and hope that works with passthough

 

If the back of the X1 connector on the motherboard is open (some are), you can use a x16 card in a X1 slot.

 

its not but i was thinking of making it open ive seen some posts about that

Link to comment

JohnM (and others),

 

I'm about to copy your design for a new unRaid setup (i currently have an unRaid setup in a tower running without virtualization).  I have a couple questions:

 

1) It looks like SuperMicro has another similar board: X9SCM-F-B.  Are you familiar with this MB at all?  Do you see any issues with using this instead of the: X9SCM-F-O?  It looks like they might be identical except the "B" version has faster onboard SATA ports?  This leads me to my next question:

 

2) As far as i can tell you're not using any of the onboard SATA ports.  If this is the case can you explain why not?

 

3) I really want to future proof my design as much as possible.  This includes adding on other virtualized unRaid servers when needed.  What SATA cards / expanders do you recommend to make this happen and how would you configure them (with the assumption that I'd just buy another Norco 4224 box when needed)?

 

Thanks in advance for your help & your wonderful & informative thread on this subject.

KMan321

Link to comment

1) The X9SCM-F-B is the bulk, non-retail, version of the same board. Both have 2xSATA3 and 4xSATA2.

2) When running unRAID as a guest VM, pass-thru of add-in controllers is easier than managing the onboard controllers and RDM.

3) The first post includes controllers and the expander.

 

Link to comment

2) As far as i can tell you're not using any of the onboard SATA ports.  If this is the case can you explain why not?

 

For me, SAS cards support SAS expanders, all 24 drives running full speed off of one card.  That is all the reason I need.

 

3) I really want to future proof my design as much as possible.  This includes adding on other virtualized unRaid servers when needed.  What SATA cards / expanders do you recommend to make this happen and how would you configure them (with the assumption that I'd just buy another Norco 4224 box when needed)?

One 8 port SAS card (IBM M1015) can support the maximum number of drives unRAID can handle using a SAS expander.  So, one IBM M1015 SAS card per unRAID VM guest attached up to a slave 4224 with a Chenbro CK23601 SAS expander (or CK22803 if you only want 20 drives)

http://usa.chenbro.com/corporatesite/products_cat.php?pos=37

 

Configuring them would consist of passing your new M1015 card through to your new unRAID guest.  That would be about it.

 

Link to comment

Perfect, thanks guys.

 

Couple more questions/comments:

1) Strange that this is the bulk version since it seems to cost more everywhere i look.  Looking at a comparison on the Provantage site it showed the "B" version had SATA300 & SATA600 ports while the "O" version had SATA150 & SATA300 ports, but of course this could be a typo.

2) This makes sense and is what i thought.

3) So, if i was looking at virtualizing 2 unRAID servers: I would install (2) M1015's in the "master" unRAID box (4224) and then one CK23601 in the "master" box and one CK23601 in the "slave" box (another 4224)?  And i'd only have to run one cable between them?

Link to comment

3) So, if i was looking at virtualizing 2 unRAID servers: I would install (2) M1015's in the "master" unRAID box (4224) and then one CK23601 in the "master" box and one CK23601 in the "slave" box (another 4224)?  And i'd only have to run one cable between them?

 

Two cables.  You will also need a dual port SFF-8087 to SFF-8088 converter.

http://www.pc-pitstop.com/sas_cables_adapters/AD8788-2.asp

 

The CK23601 has 8 external and 28 internal ports.  In the slave 4224 it works out perfectly, 8 external ports in and 24 of 28 internal ports to drives.  In the master 4224 you will have 8 internal ports in and 20 internal ports out.  The other 4 bays in your master 4224 can be hooked up to a different controller (onboard SATA if supported will work fine, or a hardware RAID card) for ESX datastores.

 

The Chenbro 6gb SAS expanders use the LSI 6gb SAS chipset, same as the Intel RES2SV240 expander.  The other chipset SAS expanders, such as the HP expander, have quirks that make them a little less desirable.

 

Link to comment

On the master box couldn't you just run a cable outside the box to the external connector on the CK23601 to get 28 port capability internally?

 

Also, just to verify...I shouldn't have any problem using the M1015 & CK23601 in the latest v5 beta build of unRaid?

 

Yes.  But...what would you install the VM guests on if all 24 drives are dedicated to unRAID?

 

The M1015 has been used quite a lot.  I know JohnM tested the Intel RES2SV240 expander which uses the same chipset as the Chenbro, but I'm not sure if anyone has tested the Chenbro with unRAID yet.

Link to comment

Has anyone tried ESXi 5.0's ability to use an SSD as a cache for memory ballooning.  This RAM still isn't in at any store and I'm looking for other options.  I know it wouldn't replace RAM completely but will it help a lot?  Can I over commit more?

 

Also, John.....that PCI bracket that held a 3.5" and 2.5"....were you ever able to purchase that?  I can't click on purchase.  Seems like a good solution to add some ESXi drives. 

 

http://www.scythe-usa.com/product/acc/064/slotrafter_detail.html

Link to comment

Has anyone tried ESXi 5.0's ability to use an SSD as a cache for memory ballooning.  This RAM still isn't in at any store and I'm looking for other options.  I know it wouldn't replace RAM completely but will it help a lot?  Can I over commit more?

 

Also, John.....that PCI bracket that held a 3.5" and 2.5"....were you ever able to purchase that?  I can't click on purchase.  Seems like a good solution to add some ESXi drives. 

 

http://www.scythe-usa.com/product/acc/064/slotrafter_detail.html

 

The slotrafter only holds 2.5" drives. I have two, I like them, they are plastic, but it works for what you need.

 

There's another product you can look into, it only holds one, but it's nice. SYBA SY-MRA25018

 

Another choice instead of using an ssd is to use a PCI based ram drive. The gigabyte i-RAM you can find them used on eBay.  Not sure it will be as fast as a modern SSD on a 6GB/s sata port, but it's a real ram drive so you would never have to worry about wear leveling.

 

I use it in my windows 7 machine for temp and pagefile.

Link to comment

Has anyone tried ESXi 5.0's ability to use an SSD as a cache for memory ballooning.  This RAM still isn't in at any store and I'm looking for other options.  I know it wouldn't replace RAM completely but will it help a lot?  Can I over commit more?

 

Also, John.....that PCI bracket that held a 3.5" and 2.5"....were you ever able to purchase that?  I can't click on purchase.  Seems like a good solution to add some ESXi drives. 

 

http://www.scythe-usa.com/product/acc/064/slotrafter_detail.html

 

The slotrafter only holds 2.5" drives. I have two, I like them, they are plastic, but it works for what you need.

 

There's another product you can look into, it only holds one, but it's nice. SYBA SY-MRA25018

 

Another choice instead of using an ssd is to use a PCI based ram drive. The gigabyte i-RAM you can find them used on eBay.  Not sure it will be as fast as a modern SSD on a 6GB/s sata port, but it's a real ram drive so you would never have to worry about wear leveling.

 

I use it in my windows 7 machine for temp and pagefile.

 

Thx for the info.  Too bad I can't find that Slot Rafter anywhere.  It's discontinued.  I already have the SYBA one and love it. 

 

EDIT:

 

I actually have a startech

 

http://www.amazon.com/gp/product/B002MWDRD6/ref=oh_o00_s00_i00_details

 

looks like Syba and Startech get their stock from the same supplier :)

Link to comment

<snip>

Recommended Upgrades:

HBA's

  • 1) M1015
    Replace The SASLP-MV8's with IBM M1015's (LSI SAS9220-8i) about $65-$85 on ebay.

<snip>

[Warning! These cards come with an IBM raid bios, you have to re-flash them to LSI IT-mode Bios to work. you can not flash them on the 9XSCM. You need to do it on another motherboard.]

<snip>

 

Hmm, I'm not clear on which firmware to flash the LSI SAS9220-8i with.  Can someone link to it or specifiy exactly which/where it is?

 

Yes, I know about the sticky in the HardDrives and Controllers section.  It doesn't mention the SAS9220-8i while it does mention the M1015 (but they all have different chipsets).

 

Thanks.

Link to comment

About to order the parts in the next couple days.  Would these still be the preferred MB & CPU?:

 

SuperMicro X9SCM-F-O

Intel Xeon 1240

 

Also, are there any supported 8GB RAM sticks available?  If not, which 4GB sticks should i get?

 

Use these: http://www.newegg.com/Product/Product.aspx?Item=N82E16820139262&nm_mc=OTC-Froogle&cm_mmc=OTC-Froogle-_-Memory+(Server+Memory)-_-Kingston+Technology+Corp.-_-20139262

 

There are 8GB sticks but they are double++ the price. 

Example:  http://www.provantage.com/kingston-technology-kvr1333d3e9sk2-16g~7KIN91M7.htm

Link to comment

Hmm, I'm not clear on which firmware to flash the LSI SAS9220-8i with.  Can someone link to it or specifiy exactly which/where it is?

 

Yes, I know about the sticky in the HardDrives and Controllers section.  It doesn't mention the SAS9220-8i while it does mention the M1015 (but they all have different chipsets).

 

Thanks.

 

As the LSI SAS9220-8i, you just update to latest from LSI (12 from Nov2011).

Link to comment

About to order the parts in the next couple days.  Would these still be the preferred MB & CPU?:

 

SuperMicro X9SCM-F-O

Intel Xeon 1240

 

Also, are there any supported 8GB RAM sticks available?  If not, which 4GB sticks should i get?

 

There are 8GB sticks but they are double++ the price. 

Example:  http://www.provantage.com/kingston-technology-kvr1333d3e9sk2-16g~7KIN91M7.htm

 

8G UDIMMS are not available yet, the above link reads special order and Kingston says;

Availability: Approx:4/18/12

Link to comment

About to order the parts in the next couple days.  Would these still be the preferred MB & CPU?:

 

SuperMicro X9SCM-F-O

Intel Xeon 1240

 

Also, are there any supported 8GB RAM sticks available?  If not, which 4GB sticks should i get?

 

There are 8GB sticks but they are double++ the price. 

Example:  http://www.provantage.com/kingston-technology-kvr1333d3e9sk2-16g~7KIN91M7.htm

 

8G UDIMMS are not available yet, the above link reads special order and Kingston says;

Availability: Approx:4/18/12

 

Try here:  Memory4less

 

 

But it doesn't really matter if they are out of stock in some places unless you have a high pain threshold because they cost so much.  8GB Unregistered ECC sticks haven't been used much so they aren't produced in large enough quantities to get the cost down.  In about a year you will see them much cheaper.  Intel seems to like creating a demand for expensive memory.

Link to comment

About to order the parts in the next couple days.  Would these still be the preferred MB & CPU?:

 

SuperMicro X9SCM-F-O

Intel Xeon 1240

 

Also, are there any supported 8GB RAM sticks available?  If not, which 4GB sticks should i get?

 

There are 8GB sticks but they are double++ the price. 

Example:  http://www.provantage.com/kingston-technology-kvr1333d3e9sk2-16g~7KIN91M7.htm

 

8G UDIMMS are not available yet, the above link reads special order and Kingston says;

Availability: Approx:4/18/12

 

Try here:  Memory4less

 

 

That website is still 2011 Christmas motif and it says; Availability: Available for Order

 

good luck getting any

Link to comment

ok, i'm gonna stick with the 4GB sticks then.  How about the other parts, are there better replacements (with regards to performance and compatibility)?  Or are these about as good as it gets for price/performance/compatibility?

 

I'm using Crucial 4 x 4GB 240-Pin DDR3 SDRAM ECC Unbuffered DDR3 1333 (PC3 10600) Server Memory Model CT51272BA1339 on my X9SCM-F-O with Xeon E1230 (ESXi 5 and unRAID + some other guests).

 

http://www.newegg.com/Product/Product.aspx?Item=N82E16820148382

 

I have started with 4GB stick adding 4GB every other week. Now I have 16GB. This MB works with 4GB, 8GB, 12GB without any issue for me.

 

Link to comment

So I'm thinking I might order (3) M1015 cards currently instead of using an expander.  I'm not positive I will be adding another unRaid server in the future so I'd like to get the best performance possible and save a little money. If I wanted to add another unRaid guest in the future I could still go this route by using (2) of the M1015 cards that i buy now and buying (2) of the chenbro CK23061 (or similar), this way i only lose one of the M1015 cards (which i could use as a backup).  How hard would it be to change my configuration in my first unRaid to support this new configuration down the road if I have a ton of drives with data in it?

 

Also, will there be any issues with the 4x PCI-E limitation on the 3rd slot on the X9SCM-F-O for the 3rd M1015 card?

 

Thanks for everyone's input so far.

Link to comment

Hey Johnm,

Remember your "lovely girl friend's sock" that you had stuffed in one of the empty drive bays? Right now, on my 4220, I only have 7 drives. Should I too use similar device to maximize airflow over the hard drives by blocking airways in empty bays? I am currently preclearing 2 2Tb drives -- a Hitachi coolspin and a WD EURS drive - and the temp during preclearing was always between 42-44 degree C. Right now, the top of the case is open since I am still waiting on my M1015 cards to arrive over the weekend. I will then close the case, tape all the holes in the sides, etc.

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.