ATLAS My Virtualized unRAID server

[JimmyJoe]

Success.....

 

Rebuilding in my parity drive right now at 78MB/Sec. 

 

So far everything is looking good. 

 

Sweet!  Congrats!  That's really good to hear. 

[jesseasi]

jesseasi,

 

I'm looking to build an unRAID server similar to yours and others using the Supermicro board and either the m1015 or the MV8 cards.  I'd like to get a Xeon Ivy Bridge CPU since it's a little lower power, but I'm a little nervous after reading this board and the issues that people have had.  You said you don't think it matters.  If you ever get a chance, I'd like to hear whether the Ivy Bridge actually works, then I'll spec it for my build.

 

Thanks,

Jesse

 

Seems that people reported issues with the Ivy Bridge.  At one point I was almost certain that was my problem.  But after swapping CPU's I still had the same problems.  I think the issue for me were the two Adaptec 1430 cards and/or maybe the brand new MV8 that I had bought.

 

I was able to return my Ivy Bridge CPU (bought it at Amazon).  I think I paid about $340 for the Ivy Bridge 3.2.  But for $50 more I was able to get a Sandy Bridge 3.4 from MicroCenter.  For the price I will go with the 80 watt increase in power consumption but the added speed.  I plan to run 4-5 different VMs with some video transcoding which will be a cpu hog. 

 

I think Ivy Bridge is ok to use. 

[lazant]

jesseasi,

 

I'm looking to build an unRAID server similar to yours and others using the Supermicro board and either the m1015 or the MV8 cards.  I'd like to get a Xeon Ivy Bridge CPU since it's a little lower power, but I'm a little nervous after reading this board and the issues that people have had.  You said you don't think it matters.  If you ever get a chance, I'd like to hear whether the Ivy Bridge actually works, then I'll spec it for my build.

 

Thanks,

Jesse

 

I can confirm I'm using a Xeon 1240 V2 (Ivy Bridge) without issue.

[kegler]

No problems here either with the Ivy.

 

I think as long as you avoid using PCIe cards known to have problems with it or combinations known not to work with it, you should be okay.  This forum is priceless as a source of information.

[Hesse34]

That's great news about the ivy bridge.  Thanks everyone for responding.

 

How about a followup question? What do you think of the SAS2-MV8 cards vs the m1015s?  I'd rather not have to mess with eBay and flashing if I don't have to.  But I've also been reading about people having problems with the MV8 cards and rc11 and parity check speeds with that mobo.  Your speeds seem pretty good, so maybe there's some hope.

 

Sent from my Nexus 7 using Tapatalk HD

[JimmyJoe]

That's great news about the ivy bridge.  Thanks everyone for responding.

 

How about a followup question? What do you think of the SAS2-MV8 cards vs the m1015s?  I'd rather not have to mess with eBay and flashing if I don't have to.  But I've also been reading about people having problems with the MV8 cards and rc11 and parity check speeds with that mobo.  Your speeds seem pretty good, so maybe there's some hope.

 

Sent from my Nexus 7 using Tapatalk HD

I really like the M1015.  I haven't used the MV8 cards so someone else can speak to those.  MV8's you do need to hack to get them to work with ESXi (see first page of this thread for details).

 

I flashed 3 M1015's yesterday with P15, two with IT and one with IR.  I used the download, usb stick and flash instructions from here and it really wasn't that hard.  I did have to try a few PC's until I found one that worked, but then it was a piece of cake.  The MB I used was a Gigabyte P55M-UD4.  Granted, it's only been about 36 hours that I have been running all 3 in my system, but not a single issue and rock solid so far and I have moved around a few TB of data.  I would hands down buy them again, just because they worked and gave me no issues.  In my unRAID VM I completed a parity sync today and have a parity check running right now with two M1015's passed through.  I am getting ~70MB/s with 9 slower drives that are all 4-6 years old.  I'm very happy with that.

 

So yes, with the M1015's if you want to pay ~$100 you are messing with ebay.  I bought mine from this seller and they are refurbished.  I couldn't tell the difference from new.  Fast shipping and great communication.  YMMV.  I did need to get high profile brackets that fit my case.  I got my brackets here.  You can get M1015's for ~$125-$140 from various retailers too.  Yes, you need to flash them.  Or you can pay a ~$250 and get a new LSI 9211-8i with full warranty.

 

 

[Hesse34]

I really like the M1015.  I haven't used the MV8 cards so someone else can speak to those.  MV8's you do need to hack to get them to work with ESXi (see first page of this thread for details).

 

I flashed 3 M1015's yesterday with P15, two with IT and one with IR.  I used the download, usb stick and flash instructions from here and it really wasn't that hard.  I did have to try a few PC's until I found one that worked, but then it was a piece of cake.  The MB I used was a Gigabyte P55M-UD4.  Granted, it's only been about 36 hours that I have been running all 3 in my system, but not a single issue and rock solid so far and I have moved around a few TB of data.  I would hands down buy them again, just because they worked and gave me no issues.  In my unRAID VM I completed a parity sync today and have a parity check running right now with two M1015's passed through.  I am getting ~70MB/s with 9 slower drives that are all 4-6 years old.  I'm very happy with that.

 

So yes, with the M1015's if you want to pay ~$100 you are messing with ebay.  I bought mine from this seller and they are refurbished.  I couldn't tell the difference from new.  Fast shipping and great communication.  YMMV.  I did need to get high profile brackets that fit my case.  I got my brackets here.  You can get M1015's for ~$125-$140 from various retailers too.  Yes, you need to flash them.  Or you can pay a ~$250 and get a new LSI 9211-8i with full warranty.

 

Thanks for that.  I was looking at the X9SCM-F mobo, but then I saw a post that mentioned the updated version X9SCM-IIF.  I think that's what you're using from what I saw in a post a few pages back.  With that mobo and ivy bridge, it should be good.  Then I'm leaning toward getting the m1015's right now just because of issues I've read about on this forum with the SAS2-MV8, Ivy Bridge, and the supermicro mobo.  I'll keep an eye on eBay to see if I can score a good price on the cards.

 

Thanks,

Jesse

[EdgarWallace]

This thread contains a lot of information.

 

Like jesseasi I discovered that my Adaptec controller (in my case the 1220SA) wasn't supported by ESXi 5.1

 

Yesterday I bought a Supermicro AOC-SAS2LP-MV8 for just 40€. Now I need some confirmation from the experts if the following steps are the ones that I need to take (I guess that the AOC-SAS2LP-MV8 is not different from the AOC-SASLP-MV8?):

• remove both Adaptec controller and disconnect unRAID drives 7 & 8 from it
  
• install the MV8 adapter and connect unRAID drives 7 & 8 to the MV8 card
  
• during boot hit "ctrl m" and disable INT 13h at the Controller Tab of the MV8 BIOS
  
• connect the SSD that I have prepared with ESXi to the MV8 and boot into ESXi
  
• create the unRAID VM and apply the MV8 hack according to the initial guide of Johnm

Question: wouldn't it be better to connect all of my 8 unRAID drives to the MV8 and just leaving the DVD drive and the SSD (datastore drive that contains ESXi with 3 VMs: unRAID, Windows 7 and Ubuntu) attached to the onboard SATA ports?

 

In that case I could use VMDirectPath for unRAID?

[JimmyJoe]

ESXi datastore performance testing - What's a good way to test and measure performance?

 

I currently have 2 datastores:

M1015 Raid1 - 2x Seagate Barracuda LP 2 TB 5900RPM SATA 3 GB/s 32 MB Cache

NFS over 1GB network to unRAID with a cache disk

 

I ran a couple of quick tests using dd like this:

dd if=/dev/zero bs=1024 count=1000000 of=<datastore>/testfile

dd if=<datastore>/testfile of=/dev/null bs=1024k

 

My quick results for a 1GB file:

Datastore    Write MB/s    Read MB/s

Local Raid1      2.9          5.2

NFS share        5.5        12.2

 

I want to test some other datastore configurations like a local drive connected to the MB, a SSD, and will probably setup ZFS/OI/napp-it with 4 drives.

 

What is a good way to test datastore performance?  Is dd good or is there something better?  Thanks!

 

[shlomiassaf]

So.. Im going ZFS as well. I think OI/NappIT

 

I got the 2nd M1015 today and Im waiting for the 16Gb (2x8) to arrive (I have 4x4)

My plan is to give 6GB to the ZFS Guest and pop it to 8GB in the future...

 

Since my rack has space for only 16drives I need to plan my setup.

 

Currently I have 2 x 1TB WD Black...

1 is an ESXi Datastore, the other is the unRaid cache + torrent store (which is terrible... but I lived with it)

I would like to use them for the ZFS Guest.

I also have a 180 GB INTEL 520 SSD and a 128Gb OCZ AGILE 3.. Both used as datastores for my main OS`s.

 

Im thinking about a backup round-trip.

All SSD`s and other VM`s backup to the ZFS.

The ZFS does a weekly backup to the unraid.

Also:

ZFS will hold unraid Cache drive.

ZFS will hold some live VM`s (OSX, testing, etc...) which will backup to the unRaid.

1 single disk, connected to the MOBO, will RDM to the download machine to be used as torrent store.

 

As I Said, I have 2 x 1TB Black drives.

I need to think ahead and I would like to keep the drive count even, the M1015 has 8 ports...

This is why I have to buy 2 more WD 1TB Blacks.

 

Now, should I use them mirroed getting 2 vdev`s of 1 TB each, 2TB total?

Or, should I use them in a RAIDZ1 getting ~3TB?

Or, RAIDZ2 for 2TB?

 

What is the actual preformance gain from mirroring?

 

Also, buying 2 drives from the same batch and putting them in the same RAIDZ increases the risk.

While in the mirroed solution i can separate them.

 

I will run guest from the pool...

 

Thanks.

 

[sideways]

A quick note for those who are wondering:

 

The Supermicro X8SI6-F and its onboard LSI SAS2008 both work great for a vsphere 5.1 based unraid box.

 

Hardware passthrough worked no problems, and I was able to flash to the IT firmware (downloaded from supermicro's site, NOT LSI's site - it makes a difference) with very few problems.

 

Thanks Johnm for documenting your build process so nicely and providing inspiration.

 

I've consolidated the hell out of my home lab and got some extra disk capacity into the bargain. Win, Win, Win.

 

I'll post a full build thread later.

[Ice_Black]

Is there a way to install Unraid on ESXi datastore without flash drive?

 

[PCRx]

Is there a way to install Unraid on ESXi datastore without flash drive?

Only the basic version. The Plus and Pro versions need the flash drive passed through to the VM.

[JimmyJoe]

Is there a way to install Unraid on ESXi datastore without flash drive?

Only the basic version. The Plus and Pro versions need the flash drive passed through to the VM.

You can also do a hybrid install.  Most everything loads from the virtual disk (fast) and the unraid mounts and uses the flash.

[peter_sm]

Is it possible to install ESXi using this guide on a USB drive and set up everything ( unraid, w7,w8,OSX ......) on VM, then remove this usb and boot on unraid USB, so I can switch between stock unraid and ESXi ?

 

//Peter

[jesseasi]

Yes that is what I have been doing.  But I also unplug the hard drives used for the ESXi VM's.  During the boot sequence you may have to hit F11 and make sure you boot from the unraid usb drive.  I don't know what would happen if you left the VM/ESXi drives plugged in.  Unraid would likely "see" them and want to set them up as available new drives.  May also detract from any testing you may be trying to do.

 

 

[peter_sm]

Good, next question, when running ESXi , is it possible the suspend (S3) the server when it's no activity on the server? like I do now when running stock unraid.

 

//Peter

[peter_sm]

I Have 8 SATA port on my MB, and one BR10i and one Adaptec 1430SA

 

On my current unraid I use totally 12 disk ,and I will not use more drives.

 

What is the best way to run unraid in ESXi ? all sata on MB and BR10i , and use the 1430SA for my datastore disk and VM ?

 

please let me know what is the best way to make progress to change to an ESXi server 

 

EDIT

 

I going to install ESXi  on a USB drive and all my datastore on SATA disk(s) than I maybe can switch between  ESXi  /stock unraid ? is that one way to go ?

 

 

//Peter

[peter_sm]

I saw now that my CPU don't support VT-D

 

What is the downside?

 

//Peter

[Wimpie]

Good, next question, when running ESXi , is it possible the suspend (S3) the server when it's no activity on the server? like I do now when running stock unraid.

 

Each VM can offcourse sleep, but this is against the design philosofie of ESXi. ESXi makes sure that each VM has as much CPU as needed, if the VM is just idling, then ESXi gives excess CPU cycles to other VM's (if they need it). Also ESXi will always run, even if all VM's are sleeping/suspended. That's just how it is, energy conservation by putting many machine on one computer, not energy conservation by switching this computer off if not needed.

 

What is the best way to run unraid in ESXi ? all sata on MB and BR10i , and use the 1430SA for my datastore disk and VM ?

 

You install unraid with the controllers in passthrough (THE reason why almost everybody uses more or less the same hardware as the Atlas build). This will ensure that you can easely change hd's. Otherwise you need to do some extra configuration steps if a drive die's in unraid. See http://lime-technology.com/forum/index.php?topic=14695.msg138466#msg138466 tip 3

 

I saw now that my CPU don't support VT-D

 

What is the downside?

 

You can't use VMDirectPath IO (i.e. pass-through hardware PCI devices directly to a VM).

 

This means that you will need to do RDM to get unraid see the drive(s). In my view, I wouldn't bother and forget ESXi with this MB/CPU.

 

 

[peter_sm]

Thanks, Since I'm going to run some OS in VM I continue using virtualbox.

 

//P

[dbezerra]

Thanks to many folks on this thread I am now running Unraid inside ESXI as well. I chose Ivy Bridge 1230, with SuperMicro X9SCM-F, 16GB of Memory and 2 IBM 1015 cards.

 

All is well, other than the 2nd LAN (Intel 82579LM). I passed it to Unraid (passthrough), but it's not being used. What are the steps to have the 2nd LAN used exclusively by UnRaid?

 

Thanks

 

[WeeboTech]

Thanks to many folks on this thread I am now running Unraid inside ESXI as well. I chose Ivy Bridge 1230, with SuperMicro X9SCM-F, 16GB of Memory and 2 IBM 1015 cards.

 

All is well, other than the 2nd LAN (Intel 82579LM). I passed it to Unraid (passthrough), but it's not being used. What are the steps to have the 2nd LAN used exclusively by UnRaid?

 

Thanks

 

 

What PSU are you using? I've been trying all day to get mine going and it seems I can't get it to boot.

[dbezerra]

Thanks to many folks on this thread I am now running Unraid inside ESXI as well. I chose Ivy Bridge 1230, with SuperMicro X9SCM-F, 16GB of Memory and 2 IBM 1015 cards.

 

All is well, other than the 2nd LAN (Intel 82579LM). I passed it to Unraid (passthrough), but it's not being used. What are the steps to have the 2nd LAN used exclusively by UnRaid?

 

Thanks

 

 

 

What PSU are you using? I've been trying all day to get mine going and it seems I can't get it to boot.

 

 

I am using a Corsair 650TXV2 Modular

[BobPhoenix]

Thanks to many folks on this thread I am now running Unraid inside ESXI as well. I chose Ivy Bridge 1230, with SuperMicro X9SCM-F, 16GB of Memory and 2 IBM 1015 cards.

 

All is well, other than the 2nd LAN (Intel 82579LM). I passed it to Unraid (passthrough), but it's not being used. What are the steps to have the 2nd LAN used exclusively by UnRaid?

 

Thanks

I don't think unRAID supports the 82579LM - but I could be wrong.  When I had mine working with ESXi 4.1/5.0 and Sandy 1230 I was using the 82579LM for ESXi with the hack and passed through the other (82574?) to unRAID.  Since I've switched to Tyan MB with all 82574(2 on one MB and 4 on other) I passed one through to unRAID with no problems.  I'm likely headed back to X9SCM-F in the next few weeks/months as one Tyan MB is flakey.