ATLAS My Virtualized unRAID server

[c3]

I suppose I should start my own thread soon.  My Norco 4224 case is here, the Tyan S5512WGM2NR mobo is on its way, and the Xeon E3 1240 is also on its way.

 

This is WAY better than XMas!

 

You got the sas version of the board? Are you going to run unRAID directly or as a guest?

 

Looking forward to your thread.

[camprman]

I did get the SAS version.  I do plan to run as a guest.

[savestheday]

I did get the SAS version.  I do plan to run as a guest.

 

This mobo is incredibly interesting....pls link to your new thread when you start it.  I love that it has the SAS2008 onboard.  That would free me up to pass thru more PCI devices to my guests.  Hmmmm....you've made me a little jealous

[Roancea]

I did get the SAS version.  I do plan to run as a guest.

 

This mobo is incredibly interesting....pls link to your new thread when you start it.  I love that it has the SAS2008 onboard.  That would free me up to pass thru more PCI devices to my guests.  Hmmmm....you've made me a little jealous

 

Yea I definitely had looked at that board as well. The one thing that occurred to me though, and maybe I'm wrong, but if you decided to go the tyan route, wouldn't it be cheaper and with possibly some advantages to going the sister model without the Integrated SAS ports?

 

Assuming you weren't in some odd position getting the SAS model for significantly less than the normal price difference between it and the non-SAS version, buying the non-sas mb + a SAS card for 60ish would net you some spare cash. The non-SAS version has an extra 8x port that wasn't dedicated to the integrated controller, so you haven't lost or gained anything there. And you're dealing with actual SAS connections on the card so if you did want to break out an expander, it'd probably be easier, not to mention cleaner than having 14 sata ports in that one corner.

 

That being said, its still a very attractive option, amd I'm (very) slightly kicking myself on buying my x9scm, not because it's not a fantastic board, but because I'm having to screw around to fit a 16x length card in one of the 8x's (hooray for odd solutions).

[Johnm]

Different tool for a different use.

 

it is hard to compare the two boards.

 

the X9SCM is 2x 8X PCIe and 2x 4xPCIe

the S5512WGM2NR is 2x 8X PCIe (one is a 16x slot but still 8x), 2x 1x PCIe and PCI slot.

it looks like you do a bit trading in slots if you include the S5512WGM2NR  sas2008 in the setup..

For me the X9SCM (and the Tyan S5510GM3NR) is more flexible slot wise. i have no need for 1x PCIe or PCI while others might.

(For example, I use hardware RAID6 VIA an Areca card on one of my X9SCM's. The SAS 2008 would just be using electricity not and doing anything)

 

Size..

The X9SCM (and the Tyan S5510GM3NR) are mATX

The S5512WGM2NR is Full ATX. that needs a full sized chassis or server chassis.

 

onboard toys..

the Tyan S5512WGM2NR wins here..

the Tyan with LSI sas 2008 and 2 USB's (VS. 1 USB on the smaller boards).

the extra USB should be handy for a recovery or back up boot drive or 1 for esxi and 1 for unraid.

keep in mind the LSI is flashed with IR firmware (I am not sure how easy it is to reflash, it should possible though).

I do dislike the lack of SFF-8087 ports. but, you should be able to expander it with a reverse breakout cable.

I do like the dual Intel 82574L on the Tyan. but with new drivers, the X9 works fine with dual NICs in ESXi.

 

cost..

The X9SCM (and the Tyan S5510GM3NR) with an M1015 should cost less.. and can be more flexible IMO

but you don't have to deal with the flashing issues. so that saves a few headaches for some people.

 

The Tyan S5512WGM2NR reminds me a lot of the Supermicro X8SI6-F. an excellent board when you can get your hands on one.

 

For me, i think the X9SCM is still a better choice, for others the Tyan S5512WGM2NR might b a logical choice.

obviously when I built this box, the Supermicro X9's where the only sandybridge servers available.

[Also keep in mind, I run 3 identical servers that do different tasks (AND OS's). having a uniform platform saves me so much time and work. by having a single platform that is more flexible is a bonus to me, to others with a single server, that is not so important. this also gives me some redundancy in a failure..]

 

I'll defiantly add the Tyan S5512WGM2NR to my list of alternate builds as soon i find that the SAS 2008 can be passed through.

 

In looking at this board, i also learned that Tyan boards do not support WOL. you must IPMI into them to wake them.

 

[camprman]

I did get the SAS version.  I do plan to run as a guest.

 

This mobo is incredibly interesting....pls link to your new thread when you start it.  I love that it has the SAS2008 onboard.  That would free me up to pass thru more PCI devices to my guests.  Hmmmm....you've made me a little jealous

 

Yea I definitely had looked at that board as well. The one thing that occurred to me though, and maybe I'm wrong, but if you decided to go the tyan route, wouldn't it be cheaper and with possibly some advantages to going the sister model without the Integrated SAS ports?

 

Assuming you weren't in some odd position getting the SAS model for significantly less than the normal price difference between it and the non-SAS version, buying the non-sas mb + a SAS card for 60ish would net you some spare cash. The non-SAS version has an extra 8x port that wasn't dedicated to the integrated controller, so you haven't lost or gained anything there. And you're dealing with actual SAS connections on the card so if you did want to break out an expander, it'd probably be easier, not to mention cleaner than having 14 sata ports in that one corner.

 

That being said, its still a very attractive option, amd I'm (very) slightly kicking myself on buying my x9scm, not because it's not a fantastic board, but because I'm having to screw around to fit a 16x length card in one of the 8x's (hooray for odd solutions).

 

I got the MoBo on sale, so that is what really put it over the edge.  I had been going back and forth, but with the sale price, it was a no-brainer. 

[camprman]

Different tool for a different use.

 

it is hard to compare the two boards.

 

the X9SCM is 2x 8X PCIe and 2x 4xPCIe

the S5512WGM2NR is 2x 8X PCIe (one is a 16x slot but still 8x), 2x 1x PCIe and PCI slot.

it looks like you do a bit trading in slots if you include the S5512WGM2NR  sas2008 in the setup..

For me the X9SCM (and the Tyan S5510GM3NR) is more flexible slot wise. i have no need for 1x PCIe or PCI while others might.

(For example, I use hardware RAID6 VIA an Areca card on one of my X9SCM's. The SAS 2008 would just be using electricity not and doing anything)

 

Size..

The X9SCM (and the Tyan S5510GM3NR) are mATX

The S5512WGM2NR is Full ATX. that needs a full sized chassis or server chassis.

 

onboard toys..

the Tyan S5512WGM2NR wins here..

the Tyan with LSI sas 2008 and 2 USB's (VS. 1 USB on the smaller boards).

the extra USB should be handy for a recovery or back up boot drive or 1 for esxi and 1 for unraid.

keep in mind the LSI is flashed with IR firmware (I am not sure how easy it is to reflash, it should possible though).

I do dislike the lack of SFF-8087 ports. but, you should be able to expander it with a reverse breakout cable.

I do like the dual Intel 82574L on the Tyan. but with new drivers, the X9 works fine with dual NICs in ESXi.

 

cost..

The X9SCM (and the Tyan S5510GM3NR) with an M1015 should cost less.. and can be more flexible IMO

but you don't have to deal with the flashing issues. so that saves a few headaches for some people.

 

The Tyan S5512WGM2NR reminds me a lot of the Supermicro X8SI6-F. an excellent board when you can get your hands on one.

 

For me, i think the X9SCM is still a better choice, for others the Tyan S5512WGM2NR might b a logical choice.

obviously when I built this box, the Supermicro X9's where the only sandybridge servers available.

[Also keep in mind, I run 3 identical servers that do different tasks (AND OS's). having a uniform platform saves me so much time and work. by have a single platform that is more flexible is a bonus to me, to others with a single server, that is not so important]

 

I'll defiantly add the Tyan S5512WGM2NR to my list of alternate builds as soon i find that the SAS 2008 can be passed through.

 

In looking at this board, i also learned that Tyan boards do not support WOL. you must IPMI into them to wake them.

 

JimWhite is using this board for his esxi server and passing through to Unraid.  He did have to put the SAS into IT mode. 

 

Definitely not saying one is better than another, just loved the price I got, and the options I will now have. 

 

 

[Johnm]

So Johnm (and everyone else).  Got that RAM mixed with 8GB of Kingston stuff.  So far so good.  24GB

 

Good info. Hynix makes excellent chips so it sounds like it might be time for me to upgrade to 32GB.

the price is still a bit high. i know it will be half that price in 6-9 months or so.

 

From what i can tell, the kingston i'll be taking out can be reused in my other servers.. so i wont be tossing it out..

2 sticks will go into my HP N40L to max it at 8GB. while the specs on kingston.com don't match. people are saying it works without issue after newegg sold them as a super combo.

the other 2 sticks can go into Goliath to bring it to 16GB.

 

just the $600 price tag for 32GB of ram is a bit more then i want to bite off.. decisions... decisions... decisions....

If i do this, i'll probably also RAID0 4xSDD's on my spare M1015 for the datastore at the same time.

[ldasilva]

 

 

If i do this, i'll probably also RAID0 4xSDD's on my spare M1015 for the datastore at the same time.

 

what size ssds are you looking at?

[Johnm]

Most likely 4x 128.

 

I have a bunch of vertex3's, agility3's and solid3's. The hard part is I only have about 2-3 of each type. I'll have to pick up at least one more and get creative on a data migration path.

 

That is unless I can mix-n-match those units. Then I have to buy none.

 

I was looking at some test results using the m1015 and 4x solid3's in raid 0. All I can say is wow.

 

I will admit. Raid 10 is not out of the question... But raid 5 is more cost effective.

 

Sent from my iPhone using Tapatalk

[Johnm]

Whelp...

 

I finally had to down Atlas...

She ran for 2 months flawlessly without any administration or maintenance.

Last day it was downed was Jan 8th to install the new expander....

not so bad. I am sure I could have kept going.. the dust was getting bad though.

 

I ran out of storage space so I inserted my last 2 "pre-drive-shortage" 3TB drives...

(I am going to miss $109 3TB Hitachi drives...)

 

They didn't come up in unraid.. doh.

I rebooted just the unraid side.. nope.. no new drives.

She was not having it..

 

I needed to clean the dust out of it as i had mentioned a while back.. today was a good day for that.. i powered it down and blew it out. it was pretty nasty in there.

on reboot, my 2 new drives came online... so much for "hot plug" or "warm plug" today.

I only had to power it off once before when adding drives. that was when I moved several drives around and added several more.

 

I also added the new "simple features" as long as i needed to reboot.

I did not do any cable management tho.. I should have.. but i didn't..

 

I also found out why one of my Hitachi's was running hot, i forgot to open the vents.. oops..

(I sort of suspected this, but I was out of town so i could not pull the drive to check. it never got over 37 while the rest were at 25-30ish)

 

It needed the spring cleaning anyways..

lets see if we can go another 2+ months..

(I am sure I'll do some upgrades before then)

 

[Johnm]

Just an update. After my weekend adventure. The unraid guest is having issues.

 

I am am getting massive problems  with shares dropping or timing out.

I am getting lower memory area errors on my console.

 

It could be the new simple features (not that likely) it could be the fact I have a preclear running (more likely). It could be a combo.

 

I totally forgot I am still running all 14 drives though a single channel of an m1015. That includes the ssd cache and parity.

I think the addition of the preclear on the same port is to much. I'll let it finish up the preclear and do some troubleshooting tomorrow.

 

Afterwards, I'll run another pass on the other port and report back.

 

Maybe it's hardware related.

 

Another possible factor is I added several hundred thousand more files to the server last week and I'm cache_dir checking it.

 

 

 

 

 

Sent from my iPhone using Tapatalk

[brian89gp]

I was running a pre-clear of 3 drives at once, each on their own channel.  unRAID eventually was unusable about 2 days in and I had to reboot it, even canceling the pre-clear tasks didn't clear up the issues.  It doesn't seem to be a bandwidth usage thing, more so a problem that comes up after a certain amount of time.  If I had to guess I would say that is what that is.

[Johnm]

I have precleard 4 drives at once on this box before I put dropped the ram down and added the expander.

 

I probably needed the system.

Goliath does not experience Any slowdown or issues while doing multi preclears. It is something in the current config on atlas.

 

I'll try again tomorrow on the non-expanded port.

 

I'm not bothered by it because I have several other servers I could run the preclear on.

Even another unraid on a spare  m1015 or mv8. I only reported it so others that copy my build might run into it.

 

And that I'll see why it happens so others don't have issues. Unfortunately I made several changes the last few months that can cause this behavior.

 

 

 

 

Sent from my iPhone using Tapatalk

[Johnm]

I decided I wanted to try Windows 8 Preview copy on ESXi today...

DOH! it does not work!!

 

I had to update my ESXi 50 to the current version.

 

I downloaded the patches from here

http://www.vmware.com/patchmgr/findPatch.portal

 

I used these instructions up to and including step5 (after that, the commands change for ESXi5.0)

http://lime-technology.com/forum/index.php?topic=14695.msg152540#msg152540

 

I followed SOME of the steps on this page. [i did steps 1,2,3. I then continued step 10 once for each patch then 11, 12 and 13]

http://communities.vmware.com/people/vmroyale/blog/2011/09/15/updating-esxi-5--single-use-esxcli-how-to

 

 

Command line I used

 

 

What You will see in your Event log so you know it is working. Note the location I stored my Updates in vs. the CMD i ran.

Also note the output from the CMD prompt when the script was done.

 

 

Windows 8 (without any hacks). Also note that my ESXi version has gone up after a reboot. it is now 5.0.0, 515841

 

 

 

 

 

[Johnm]

Has anyone passed thru USB hubs on this mobo?  I've got the unRAID USB plugged into motherboard header and the ESXi stick plugged into a header adapter like this:

 

http://www.amazon.com/Koutech-Accessory-IO-UU220-Header-Pin-Adapter/dp/B003UIM142/ref=sr_1_10?ie=UTF8&qid=1328886569&sr=8-10

 

I really want to pass through another USB header directly to unRAID but have followed JohnM's advice about having a non working ESXi stick afterwards.  Is there and easy way find which is which, or has anyone mapped the headers to USB controllers in ESXi?  I really see this as the only solution to my APC problem (I don't get alerts when the power goes out even though the UPS is recognized in a normal USB pass through).  Easy method would be to install a USB card but currently my only free slot is taken up by a firewire 800 card passed thru to Mac OS X Lion.

 

as far as i can tell

usb controller 2 is the internal and usb controller 1 is external(the ones on the rear I/O).

Not sure about the solo internal port

[Rajahal]

Not sure if this will help any of you, but this 16 GB server RAM kit has recently arrived at Newegg:

 

Kingston 16GB (2 x 8GB) 240-Pin DDR3 SDRAM ECC Registered DDR3 1066 Server Memory QR, x8 w/Therm Sen Model KVR1066D3Q8R7SK2/16G

 

According to Kingston it is compatible with the Supermicro X8SIL-F-O, so it probably works with some of the newer Supermicro boards as well.  $150 seems quite a bit cheaper than other options I've seen in this thread, but it is relatively 'slow' RAM at 1066.

[Johnm]

Not sure if this will help any of you, but this 16 GB server RAM kit has recently arrived at Newegg:

 

Kingston 16GB (2 x 8GB) 240-Pin DDR3 SDRAM ECC Registered DDR3 1066 Server Memory QR, x8 w/Therm Sen Model KVR1066D3Q8R7SK2/16G

 

According to Kingston it is compatible with the Supermicro X8SIL-F-O, so it probably works with some of the newer Supermicro boards as well.  $150 seems quite a bit cheaper than other options I've seen in this thread, but it is relatively 'slow' RAM at 1066.

 

Nice!

strange that I did not get my email notification that they were in. I think I'll pick up a kit this week. I was going to order the hynix chips, but this will do fine.

[Johnm]

Just an update. After my weekend adventure. The unraid guest is having issues.

 

I am am getting massive problems  with shares dropping or timing out.

I am getting lower memory area errors on my console.

 

It could be the new simple features (not that likely) it could be the fact I have a preclear running (more likely). It could be a combo.

 

I totally forgot I am still running all 14 drives though a single channel of an m1015. That includes the ssd cache and parity.

I think the addition of the preclear on the same port is to much. I'll let it finish up the preclear and do some troubleshooting tomorrow.

 

Afterwards, I'll run another pass on the other port and report back.

 

Maybe it's hardware related.

 

Another possible factor is I added several hundred thousand more files to the server last week and I'm cache_dir checking it.

 

 

UPDATE TO THIS:

I split the expander again.

I am back on my Method #2

4 full speed drives on the first ports and 20 drives Split on the second port with the expander.

 

I am preclearing my 7200 RPM 3tb drive on one of the fullspeed ports and streaming 2 1080p blurays and no lag or slowdown.

The 3TB is preclearing pretty fast but is very warm. 43 deg. that's still within limits. I just need to keep an eye on it during my first sync.

 

I'm glad it is all back to normal.

[c3]

Not sure if this will help any of you, but this 16 GB server RAM kit has recently arrived at Newegg:

 

Kingston 16GB (2 x 8GB) 240-Pin DDR3 SDRAM ECC Registered DDR3 1066 Server Memory QR, x8 w/Therm Sen Model KVR1066D3Q8R7SK2/16G

 

According to Kingston it is compatible with the Supermicro X8SIL-F-O, so it probably works with some of the newer Supermicro boards as well.  $150 seems quite a bit cheaper than other options I've seen in this thread, but it is relatively 'slow' RAM at 1066.

 

This will NOT work in the x9 series (X9SCM, etc). They require unbuffered.

[Rajahal]

Not sure if this will help any of you, but this 16 GB server RAM kit has recently arrived at Newegg:

 

Kingston 16GB (2 x 8GB) 240-Pin DDR3 SDRAM ECC Registered DDR3 1066 Server Memory QR, x8 w/Therm Sen Model KVR1066D3Q8R7SK2/16G

 

According to Kingston it is compatible with the Supermicro X8SIL-F-O, so it probably works with some of the newer Supermicro boards as well.  $150 seems quite a bit cheaper than other options I've seen in this thread, but it is relatively 'slow' RAM at 1066.

 

This will NOT work in the x9 series (X9SCM, etc). They require unbuffered.

 

Good to know, thanks.

[Johnm]

Lol

 

that might be why I didn't get the email stating the correct memory was in stock.

 

Thanks for the update c3. I didn't click the link yet.

 

 

Sent from my iPhone using Tapatalk

[Johnm]

Just bought 16GB of this RAM

 

http://www.superbiiz.com/detail.php?name=W1333EB8GH

 

this stuff just dropped about $30 per stick in the last day or 2...

$122.99 a stick (minus another $15 with promo code "pinch")

full 32GB for $500.. still a bit high.. but better then the $600 it was last week.

 

I bet it will be like $100 a stick in 2-3 months.

 

 

    Specifications

        Mfr Part Number: W1333EB8GH

        Type: DDR3

        Capacity:Â 8 GB

        Speed: PC10600 1333MHz

        Size & Bit: N/A

        Pins: 240pin

        ECC: Yes

        Registered: No

        Chipset: Hynix

 

 

EDIT:

mehh got 4 sticks... hope it works

[savestheday]

I should have posted an update!  I've got 16GB of the Hynix stuff running alongside the Kingston 8GB. All has been running perfect for 2-3 weeks now. You should be fine John. Tempted to pick up another 16GB at that price.

[Johnm]

I should have posted an update!  I've got 16GB of the Hynix stuff running alongside the Kingston 8GB. All has been running perfect for 2-3 weeks now. You should be fine John. Tempted to pick up another 16GB at that price.

 

Good to know..

 

then I got this today..

 

We are sorry to inform you that the item you ordered, ( Super Talent Server Memory, Part # W1333EB8GH), is currently out of stock, but the good news is that we are expecting to receive new shipments in 10 days. I apologize for the delay.

Oh well.

Maybe it will be cheaper in 2 weeks and I get a price reduction.