ATLAS My Virtualized unRAID server

kaiguy · February 11, 2012

I actually just bought a PCI-Express USB 3.0 card, with 3 external ports and one internal port. I plan to pass it to my unRAID VM, hoping I can then get full speeds for the flash drive and any external USB drives I may wish to use. I'll report back with my results, as I haven't read about anyone here doing that (but there are quite a few posts I may have missed). For $20, I thought it was a fair gamble in hopes it will work.

Johnm · February 11, 2012

I actually just bought a PCI-Express USB 3.0 card, with 3 external ports and one internal port. I plan to pass it to my unRAID VM, hoping I can then get full speeds for the flash drive and any external USB drives I may wish to use. I'll report back with my results, as I haven't read about anyone here doing that (but there are quite a few posts I may have missed). For $20, I thought it was a fair gamble in hopes it will work.

Thats a nice price for all it has to offer.

I am just afraid it will only work as a USB 2.0 card without the drivers (or not at all). I am not sure if they are in the latest beta's or not?

Let us know how that works and what unRAID version(s) you tested with.

kaiguy · February 11, 2012

I read that the Linux kernel supported USB 3.0 since 2.6.31... so I would imagine unRAID would have it?

Unfortunately, however, ESXi just doesn't want it to work. First it made my MV8 disappear. Then when I rearranged my cards, it did show up under the advanced configuration as "Unknown USB." However, for the VM passthrough it was being displayed as "Unknown Unknown." Ultimately I was unable to add any USB devices connected to it in the VM configuration. Shame. I bet I would have had better luck if I got a card with an NEC controller... my best guess is that that the cheapo VIA controller is the issue.

It was worth a shot.

publicENEMY · February 11, 2012

SATA Expansion Card(s):
2 X SUPERMICRO AOC-SASLP-MV8 $109.99 each
[This will only get you 16 drives in unRAID. you will need 3 for up to 24. Se recomended hardware below.]

Pardon me for asking(first time building a server). if i want 20 unraid drives, how many of SUPERMICRO AOC-SASLP-MV8 should i get? i thought supermicro mb have on board sata connector?

Cables:
4 x NORCO C-SFF8087-D SFF-8087 to SFF-8087 Internal Multilane SAS Cable $19.99 Each Newegg
2 x NORCO C-SFF8087-4S Discrete to SFF-8087 (Reverse breakout) Cable $14.99 Each Newegg
1 x NORCO C-P1T7 4Pin Molex 1 Male to 7 Female Power Extension Splitter $7.99 Newegg

since i heard that norco customer service is bad and have lots of faulty backplane etc, i want to buy cable for norco from supermicro. problem is, i cant find the equivalent pars from supermicro. i am total nood.

c3 · February 13, 2012

SATA Expansion Card(s):
2 X SUPERMICRO AOC-SASLP-MV8 $109.99 each
[This will only get you 16 drives in unRAID. you will need 3 for up to 24. Se recomended hardware below.]

Pardon me for asking(first time building a server). if i want 20 unraid drives, how many of SUPERMICRO AOC-SASLP-MV8 should i get? i thought supermicro mb have on board sata connector?

Cables:
4 x NORCO C-SFF8087-D SFF-8087 to SFF-8087 Internal Multilane SAS Cable $19.99 Each Newegg
2 x NORCO C-SFF8087-4S Discrete to SFF-8087 (Reverse breakout) Cable $14.99 Each Newegg
1 x NORCO C-P1T7 4Pin Molex 1 Male to 7 Female Power Extension Splitter $7.99 Newegg

since i heard that norco customer service is bad and have lots of faulty backplane etc, i want to buy cable for norco from supermicro. problem is, i cant find the equivalent pars from supermicro. i am total nood.

This is an ESX host build, you may wish to investigate Goliath, which is unRAID and uses motherboard ports.

savestheday · February 14, 2012

I ended up buying this card:

http://www.amazon.com/gp/product/B0024JR2CO/ref=oh_o00_s00_i00_details

Going to pass thru the firewire to the Mac OS X guest and the USB to unRAID.

kaiguy · February 14, 2012

I ended up buying this card:

http://www.amazon.com/gp/product/B0024JR2CO/ref=oh_o00_s00_i00_details

Going to pass thru the firewire to the Mac OS X guest and the USB to unRAID.

I hope you have better luck than I did, but I'm pretty sure not going with an NEC controller was my problem. Please keep us posted on how it works out!

Edited: Actually, considering it's USB 2.0, I think you're def good to go. Nice find.

PS. How difficult was it to get an OS X VM?

publicENEMY · February 15, 2012

SATA Expansion Card(s):
2 X SUPERMICRO AOC-SASLP-MV8 $109.99 each
[This will only get you 16 drives in unRAID. you will need 3 for up to 24. Se recomended hardware below.]

Pardon me for asking(first time building a server). if i want 20 unraid drives, how many of SUPERMICRO AOC-SASLP-MV8 should i get? i thought supermicro mb have on board sata connector?

Cables:
4 x NORCO C-SFF8087-D SFF-8087 to SFF-8087 Internal Multilane SAS Cable $19.99 Each Newegg
2 x NORCO C-SFF8087-4S Discrete to SFF-8087 (Reverse breakout) Cable $14.99 Each Newegg
1 x NORCO C-P1T7 4Pin Molex 1 Male to 7 Female Power Extension Splitter $7.99 Newegg

since i heard that norco customer service is bad and have lots of faulty backplane etc, i want to buy cable for norco from supermicro. problem is, i cant find the equivalent pars from supermicro. i am total nood.

This is an ESX host build, you may wish to investigate Goliath, which is unRAID and uses motherboard ports.

thank you for clearing up the confusion. so it means that i can just use 2 saslp and mb sata ports to get 20 drives in unraid, right?

savestheday · February 15, 2012

I ended up buying this card:

http://www.amazon.com/gp/product/B0024JR2CO/ref=oh_o00_s00_i00_details

Going to pass thru the firewire to the Mac OS X guest and the USB to unRAID.

I hope you have better luck than I did, but I'm pretty sure not going with an NEC controller was my problem. Please keep us posted on how it works out!

Edited: Actually, considering it's USB 2.0, I think you're def good to go. Nice find.

PS. How difficult was it to get an OS X VM?

It was a little difficult if you don't understand hackintoshes. Luckily I've built quite a few of those in my day Honestly tho, that's not a huge a pain since ESXi 5.0 natively supports Mac OS X on its hypervisor, you just need to make it think it's running on Mac hardware.

Here it is in a nutshell:

Patch ESXi 5.0 to remove the Mac hardware limitation, reboot. (http://www.insanelymac.com/forum/index.php?s=b26b9cfe2f95dd2dd64696cf47c161aa&showtopic=267296P)

Create a Mac OS X VM

Pass thru a Lion USB stick to that VM

Boot from that USB with some special parameters (Which I had to look up)

Install Lion and then VMware Tools from VMware Fusion

Caveats:

There are some known issues with managing the Guest via vCenter. Essentially, you can't do it. You'll have to connect directly to your host. (see link above)

Video has gotten better, but it's not perfect. (auto fit, 3D, etc)

The patch/unlock of ESXi can be completely undone without having to reformat your stick. It's just flipping a bit. Very easy to do this. Backup your stick first anyway.

I will be re-doing my Mac guest with my new RDM SSD. I can write up a tutorial for the forum if enough ppl think it's worth it.

savestheday · February 15, 2012

Will do! It's really not that hard, just need to do it from scratch to remember the steps and write them down

kaiguy · February 15, 2012

I read through some basic instructions a few weeks ago, but got deterred when I saw it patches the actual firmware. But you say it's easy enough to rollback? Might have to do it then, as I'm pretty much 100% Mac in my household (minus ESXi VMs).

savestheday · February 16, 2012

Yup, it's easy to rollback. Have you ever used PPF-O-MATIC or something similar? It's flipping bits (afaik). So if you want to remove the limitation of running on Mac hardware, it removes that(those) bit(s). If you want to go back to vanilla, it restores them. That's how I see it at least.

Update on that FW800/USB 2.0 card. USB 2.0 passthru to unRAID works but it's still not kicking off my script when the power is pulled. The search continues.

kaiguy · February 16, 2012

Update on that FW800/USB 2.0 card. USB 2.0 passthru to unRAID works but it's still not kicking off my script when the power is pulled. The search continues.

Is unRAID loading faster now?

savestheday · February 16, 2012

Update on that FW800/USB 2.0 card. USB 2.0 passthru to unRAID works but it's still not kicking off my script when the power is pulled. The search continues.

Is unRAID loading faster now?

I've been using the VMDK method for quite some time now so I didn't even check. I'm sure it'd boot pretty quickly tho. This card does have an internal USB port.

What's really crappy is I thought that I'd be able to divvy up the FW800 and USB 2.0. Not so with passthru. Since they're on the same card, the unRAID won't boot if FW800 is passed thru to Lion. It says Device is busy. They're two different chipsets too

Screen%20Shot%202012-02-16%20at%2011.43.11%20AM.png

Screen%20Shot%202012-02-16%20at%2011.42.31%20AM.png

kaiguy · February 17, 2012

I ended up getting this USB 2.0 PCI-e card from Monoprice. It's fantastic to finally get regular unRAID boot speeds.

I was also hoping it would fix the consistent COMMLOST error I get in my Windows apcupsd slave, but unfortunately it's still happening (constantly loosing its connection then reconnecting). Maybe it's a problem with my UPS or the cable. Not sure if that's similar to the issue you're getting, savestheday. I would imagine, however, with the frequent loss of connectivity that it wouldn't get the powerdown signal if the power is pulled.

savestheday · February 17, 2012

I thought about it and the only time email alerts ever worked for me was when I was using the unMENU version of the APCUPSD plugin. I uninstalled the regular plugin and went back to the unMENU version. SUCCESS! Now, I'm not sure whether it's a combination of the USB 2.0 card + the old plugin or if the old plugin would work with a normal USB port. That's what I 'm going to test next. Figured that I'd pass through the 1.5TB drive I had setup in FW800 enclosure as another RDM drive passed thru to my Lion server. The SSD RDM is FAST!

I think the difference between the old plugin and the new is that the new defaults to USB while the old defaults to SMART. I've tried using SMART on the new version and it didn't work.

kman321 · February 19, 2012

So everything seems to be running pretty good except: after streaming a blu-ray across the network from the virtualized unraid box i started getting the following errors in syslog and my network performance would tank:

Feb 19 02:15:30 Tower02 kernel: swapper: page allocation failure: order:3, mode:0x4020

Feb 19 02:15:30 Tower02 kernel: Pid: 0, comm: swapper Not tainted 3.0.3-unRAID #7 (Errors)

Feb 19 02:15:30 Tower02 kernel: Call Trace: (Errors)

Feb 19 02:15:30 Tower02 kernel: [<c105f857>] warn_alloc_failed+0xb2/0xc4 (Errors)

Feb 19 02:15:30 Tower02 kernel: [<c105ffc2>] __alloc_pages_nodemask+0x456/0x47f (Errors)

I've set unRaid to use the 2nd NIC in my SuperMicro X9SCM (Intel 82579LM using E1000 driver in VM). After setting /proc/sys/vm/min_free_kbytes = 8192 I was able to get the network performance back but I'm still seeing the above errors. Any ideas? Should I passthrough the NIC instead of trying to have the guest manage it?

kaiguy · February 19, 2012

Are you running vmtools? Just out of curiosity, why did you choose E1000 and not vmxnet3?

kman321 · February 19, 2012

I'm not running VMTools because i wasn't sure if i wanted to rely on them since they aren't officially supported in unRaid. What's are the benefits of installing them? Should I install them?

kaiguy · February 19, 2012

In addition to vmxnet3 (which is a much more efficient and faster virtual network driver than E1000), you get the ability for ESXi to gracefully shut down your unRAID guest. Also, you can mount host shared folders.

kman321 · February 19, 2012

Is that a better option than just doing passthrough on the NIC?

Shigo_Naito · February 21, 2012

Has anyone tried to run a M1015 (flashed) in ESXi without passing it to unraid? I already have one M1015 passing to unraid and i only want to move 2 more drives to unraid and have the rest for other VMs. Unfortunately i cant get ESXi 5 to see the adapter so that i can edit the drives. Meaning it sees the cards and lables them correctly but they are not avalable in the starage adapter settings and there for the drives are not being seen. I dont understand what i might be messing, any help would be great.

thx

RimBlock · February 21, 2012

Firstly, nice write-up up.

VT-d only allows the passthrough of the controlling chipset (ie the LSI 2008 on the M1015 card), not the devices connected to that chipset.

For the M1015 this means that it is all or nothing and you cannot select individual drives.

On another note, some other decent boards to look at for an ESXi build include;

Supermicro X9SCL/X9SCL-F (C202 so only SATA II and less PCIe slots but otherwise the same as the X9SCM-F).

Intel S1200BTL/S1200BTS (C204/C202 ESXi certified boards. BTL has a separate connection for a 4 port SAS controller).

Supermicro X9SCI-LN4 (C204, 4 ESXi supported lan ports on the board but limited expansion slots).

If you want to go small (rare in the world of masses of storage I know) you could look at the Intel S1200KP mITX C206 board (ECC or non-ECC ram, Dual lan - one natively supported by ESXi, PCIe x16 slot - add a storage controller maybe). Should be a fair bit cheaper than the Supermicro mITX boards. Cheaper still, the Intel DQ67EP mITX board supports VT-d and works well.

RB

Shigo_Naito · February 21, 2012

I am not looking to passthrough a drive, what I would like to do is have some RDM drives and then some datastores. But ESXi does not show the controler or the drives in the client software.

Roancea · February 21, 2012

***Apologize in advance for being long-winded, and if I should move this to a separate thread, please let me know. Just posted here as this seems to be some of the most comprehensive discussion on esxi builds on these forums.

I've been toying with doing this type of build with my unraid server for some time, and finally bit the bullet - in no small part due to the inspiration from this thread. I have a X9SCM-F-O with a E3-1230 on its way, and am looking forward to getting to tinker with this hopefully over the weekend. However, I'll be the first to say I'm very green to esxi, so hopefully you guys can help me come up with a few answers.

To start off, what I'm planning on running in this box is:

Unraid server on a passed through M1015 (relatively small at the moment, only 5 2Tb’s, no cache)

Usenet box (sabnzbd/sb/cp)

Mythtv guest (possibly mythbuntu) – recording from HDHR over IP

W764 (passed through graphics as secondary system)

Odds and ends others as I go, but probably nothing too taxing

My 2 major concerns at the moment in my planning are datastores and network management.

Datastores – unraid is going to be off the flashdrive, but as for the rest, I’m not sure the best way to go. If the W7 box works out, I may just pass through a dedicated SSD for it. That leaves the usenet and mythtv guests.

The usenet clients I doubt will take up much space in themselves, and I don’t think it will be a problem (?) to have them just download straight to the unraid share. However, my real concern comes with the Mythtv guest – I intend to have it recording a fair amount from an HDHR as well as processing the recordings to mkv’s to store on unraid. I’ve toyed with the idea of having a dedicated 1TB 7200 drive for this guest so that it can freely record/repack without interference from other VM’s.

Is this a good way to handle this? If so, what should I get to hold the VM’s themselves? Would it be viable to just have a 128GB SSD hold the Mythtv and usenet VM’s in addition to the W7 VM rather than pass through an SSD to the W7 box?

Networking – I believe I read somewhere in this thread that Johnm had originally intended on passing one of the gigE ports to unraid, but then had done fine with the bandwidth and left everything through just esxi. While my unraid usage is not very high (majority use is media, maybe 2-3 simult. streams max), my concern here is how much bandwidth will be used mostly by the mythtv VM. With a few streams coming in from the HDHR, as well as potentially streaming out via mythbox to xbmc frontends, I really have no idea how much bandwidth this may consume.

I love the idea of the internal 10G vlan, especially since the myth VM will likely be transferring a fair amount of data to unraid, but am I going to be running into problems at some point with all of this going through a single gigE port managed by esxi? If so, is there some way (not too terribly hard, or at least that I could figure my way through) to have the mythtv VM be passed the second gigE port but still have all traffic to/from the other VM’s routed internally?

ATLAS My Virtualized unRAID server

Recommended Posts

Link to comment

Top Posters In This Topic

Popular Days

Top Posters In This Topic

Popular Days

Popular Posts

Johnm

Johnm

Johnm

Posted Images

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Join the conversation