UnRAID on VMWare ESXi with Raw Device Mapping

[prostuff1]

Unfortunately I wasn't able to get the pci device pass-through working on the X7SBE tonight. VT and VT-D are both enabled in the bios so I don't know what else to do.

 

A quick search found a mailing list conversation aboutX7SBE VT-D problems in Xen:

after a few tests, i can confirm that VT-d on Supermicro X7SBE/X7SB4

doesn't work at all.

I assume this has to do with invalid ACPI table content.

 

Searching for X7SB4 & VT-D also yields a few posts about it not working with that similar board.

 

Damn!

 

I think this board would make a perfect candidate for something like this.

 

I just picked up a C2SEE and may try with that.  I also have a Gigabyte board that has 1 PCIEx16 and 4 PCIEx1 that I can try this on later.  I will have to do some searching, reading, and testing to see what works and what won't.

[MvL]

Guys can you keep me posted. I'm also interested in this setup. I want to build a new unRAID machine next year. Maybe i have to build a ESXi machine and run all my physical server on the ESXi machine.

 

A guide is indeed very nice!

[cj0r]

Thank you all for your advice so far. I could get a PCI-X mobo but I'm kinda attached to my X8ST3-F and I love the IPMI, so would rather get a PCI-E solution working. The PCI-X card is OLD also - BIOS is from 2005, so always blame it when I have issues. If the new PCI-E I ordered yesterday doesn't work then I will run 2 boxes for a while - i have an old Xeon PCI-X mobo that just needs a bigger PSU and that can be my unRAID box. I really want 1 box though, won't rest till I do.

 

I guess I could use Raw Device Mapping but don't want 8 drives spinning all the time.

 

Look forward to hearing your progress, will report mine at the weekend when I get my AOC-USASLP-L8i

 

Jon

 

Question regarding your X8ST3-F, I noticed that it has a LSI 1068E 8-Port SAS controller and the Intel ICH10R 6-Port SATA controller onboard; do you just pass those through ESXi to unRaid and still get control of spin down etc but lose the ability to use them in any other virtual machines?  Or is raw device mapping the only way to utilize those ports which leads to you losing control of spin down etc.?

 

At this point I don't really need the full 20-24 drive array... currently only have 8 drives in my server.  Just being able to use a motherboard with 14 usable ports would be nice, obviously I would need to throw in additional controller cards for use in other VM's though.

[tranm5]

I'll get an Asus P5Q-VM DO tomorrow. This board should be able to passthrough a controller card to my UnRaid VM.

 

I'll try 2 Promise card, one PCI et one PCIe

 

I'll keep you informed of the results

[nojstevens]

Thank you all for your advice so far. I could get a PCI-X mobo but I'm kinda attached to my X8ST3-F and I love the IPMI, so would rather get a PCI-E solution working. The PCI-X card is OLD also - BIOS is from 2005, so always blame it when I have issues. If the new PCI-E I ordered yesterday doesn't work then I will run 2 boxes for a while - i have an old Xeon PCI-X mobo that just needs a bigger PSU and that can be my unRAID box. I really want 1 box though, won't rest till I do.

 

I guess I could use Raw Device Mapping but don't want 8 drives spinning all the time.

 

Look forward to hearing your progress, will report mine at the weekend when I get my AOC-USASLP-L8i

 

Jon

 

Question regarding your X8ST3-F, I noticed that it has a LSI 1068E 8-Port SAS controller and the Intel ICH10R 6-Port SATA controller onboard; do you just pass those through ESXi to unRaid and still get control of spin down etc but lose the ability to use them in any other virtual machines?  Or is raw device mapping the only way to utilize those ports which leads to you losing control of spin down etc.?

 

At this point I don't really need the full 20-24 drive array... currently only have 8 drives in my server.  Just being able to use a motherboard with 14 usable ports would be nice, obviously I would need to throw in additional controller cards for use in other VM's though.

 

Drealit,

 

This may be my lack of knowledge of how to achieve this - but I had wanted to test this out - i.e. using the LSI 1068E 8-Port SAS as passthru to unRAID. The issue with this is I can't figure out in the cards BIOS how to add a drive just as a physical drive and not as part of a RAID array. The POST shows my SATA drives however. Saying that, I don't recall that i've tested just passing thru that controller with the drives set up as physical drives only - although memory tells me I did and unRAID didn't see the devices. I'm travelling today but will try this again tomorrow and report back

 

Jon

 

 

[cj0r]

Some discreet controllers I've dealt with in the past have required reflashing to bypass the RAID software so that it could be used as a normal storage controller... it's possible that this controller is acting in the same way but I've never met one that was onboard that wouldn't allow you to just simply change it to something other than RAID.  Hmmm killing all my ideas haha!

[nojstevens]

Drealit,

 

Ok - i've tried some things and no good news yet..

 

I passed thru the LSI on board RAID to unRAID in a vm and it did not see any devices, so I booted unRAID natively and again it did not see any devices. Devices do show at POST. So, next I figured I will attach my VM boot stuff and Datastore disks to the LSI SAS and attach my unRAID disks to the SATA x 6 and try to boot off SAS. Drives show at post as expected but then in the BIOS i cannot choose a LSI disk to boot from. I am thinking it will not work with single drives and needs an array. I will look for a new BIOS and try with that.

 

So. fingers crossed for my new SM card that doesn't even look like it shipped yet, even tho I paid 2 day shipping on Amazon. Time for email complaint....

 

I will get this to work... even if I bankrupt myself with restocking fees

 

Jon

[pfennigcat]

Thanks for the continuing work guys!

 

I am just doing the initial setup on my first unRaid system (see my sig).  I don't plan to go beyond 3 disks for the next year or so, and probably will never need an add-in SATA card to support new drives (ha ha ha, yeah right) so I'll be fine with just my onboard SATA.  I do want to move my zoneminder machine (ubuntu linux with a PCI 8-port analog camera capture card) into the new hardware if possible though, alongside unraid in ESXi.  I have some questions - some may have already been answered in this thread, but I'm not sure if I read it correctly or if there was updated info somewhere.

 

1) The unRaid VM doesn't have any virtual disks assigned to it - it's just a diskless VM set up with RDM to connect the USB key and the parity/data drives, right?

1a) So if things don't work out with ESXi, I can just tell my BIOS to boot from the USB key directly rather than booting the separate drive with ESXi on it, and UnRaid will work in stand-alone mode without any modifications?  Would I need to take special steps to make sure RDM disk order matches physical disk order or something?

Or to put it another way, I can set up my standard UnRaid install now, with all my data, and try out UnRaid on ESXi once I find another USB drive for ESXi without rebuilding or losing any data?  And can switch back if ESXi isn't working out for me?

1b) Because unRaid doesn't have any virtual disks, there's no real point to taking a VM snapshot, right?

 

2) I have one PCI card (camera capture for ZoneMinder VM) and a 16x PCIe card (nvidia graphics card for XBMC) that I might use on non-UnRaid VMs.  My mobo has onboard Intel GMA X4500 graphics.  Is there any conclusion about whether I will be able use these three devices on separate VMs or whether they will have to be assigned to the same VM?  It's a mix of PCI, PCIe, and intel onboard graphics, so is there a chance they can all be used independently?

 

3) I plan to use a spare 320GB disk I have for the ESXi installation and to host the VM disk images for the ZoneMinder and XBMC VMs (ZM at least will need quite a bit of space for camera recordings).  Is there any benefit to installing ESXi on a USB drive instead?  Lower power usage if I'm not using those VMs, just the unRaid VM?  Anything else?

 

4) Is it still the case that I won't be able to get drive temps or be able to spin up / spin down the RDM drives from UnRaid when in a VM?  Or was that only for drives on add-in cards?

 

Thanks!

[nojstevens]

I have great news.....

 

Drealit got me thinking the other day about using the LSI 1068E SAS on the mobo of my X8ST3-F - the issue was that neither ESXi nor unRAID could see devices attached to this controller. It would not let me use devices as physical devices, it had to make a RAID array. Well, that kinda annoyed me so googling i have been, and i found this link

 

http://ubuntuforums.org/archive/index.php/t-973912.html

 

regarding deactivating the MegaRAID bit and just using the card as a controller. Basically I had to remove a jumper on my mobo, flash the firmware and then I rebooted and SATA drives are recognized by ESXi and unRAID.

 

So, loaded my array drives in, had to do an initconfig and then remade my array. Moment of truth comes when I do a parity sync - previous cards have either crashed or gone so slow it told me 171 days to sync. The great news is on this config I am getting 65Mb/sec on my parity sync.

 

So, it would appear we have cracked it - unRAID 5 beta 2 working with physical card passthru - its taken me 2 weeks, some very late nights and a upset wife, but I finally have a solution - and I now can put all of my PC's into one physical box....

 

I can't tell you how good I feel right now.... but I think some of you probably know.

 

Thanks to you all for your help

 

Jon

[heffe2001]

I got all my parts in, and have set up the LSI card I have for passthrough.  The unraid VM I have setup sees it just fine, and sees the drive I have currently connected.  When I assign it to the array (just the one drive connected), it lets me start it, and let me format it, but it shows the drive name on the status page as __.  Under the raid configuration, it shows it as:

disk1: pci-0000:03:00.0-sas-phy0:1-0x1221000000000000:0-lun0 host1 (sdb) WDC_WD20EADS-00_WD-WCAVY5772801

 

I'm also not getting any temps from the drive, nor will it spin down.  I could probably make it work like this, but I'd really like the spindown & all that working.  This was all with 5.0b2.

 

I'm currently running a Asus M4A89TD Pro/USB3, Amd 810 Phenom II Quad, 4g DDR3, and an LSI Logic SAS3081E-S 8 Port SAS Controller, with a single 2tb WD Green drive on it.

 

 

[nojstevens]

I'm getting my temps and serial numbers. Did you look to see if there is newer firmware for your controller?

 

Jon

[heffe2001]

You know, I didn't do that...  I'll see if there is and update then report back.

 

*EDIT*

 

I flashed it with the latest version bios I could find (back from June I believe it was), in IT mode, not Raid mode, and still didn't see the temps in unraid 5b2.  Did you use raid mode, or IT mode?

 

2nd edit:

 

Just tried it in NATIVE mode, non-vm, booted off my unraid key and STILL no temps or name on the main array screen...  Not sure what's up with that.  It should be the same as your card, but for whatever reason I'm not getting the temp info in unraid.  Wonder if it could be a 5b2 problem with my machine?

 

 

 

 

 

[nojstevens]

Hi heffe2001,

 

I am using IT mode - this is the firmware I flashed with...

 

ftp://ftp.supermicro.com/driver/SAS/LSI/1064_1068/IT/Firmware/B3/L4i/1.26.01/

 

Good luck

 

Jon

 

[heffe2001]

Looks like those drivers are for a different series (at least the firmware installer says so anyway).  The drivers I'm using are from October of this year.  I've tried the IT & Raid versions, same thing on both.  The name of the drive always shows as __, and 0c for the temps.  Same drive plugged into the mobo ports shows correctly with the temp & name.  I'm really at a loss, as the system appears to see the drive names correctly when you assign the drives to the port, but the DISPLAY doesn't show any name/serials.  THe problem there is, if you plug say a different drive into a specific port and config it, then plug a different drive into that port when you reboot, unRaid says its the correct drive in that slot, even though it's definitely not (swapped a 160g out for a 2tb, and unraid happily said the array was fine).

 

I'm wondering if it's a kernel thing, might need some extra support for my specific card or something.  I even went back to the latest 'stable' release and the same results..

 

[nojstevens]

When you assign them in the assign menu, do you see the serial number and size then?

 

Are you using WD or Seagate? The reason I ask is I'm almost all WD but I have a Seagate and that drive does not show temp and serial # on this card, but on SATA it does. And in the assign screen, I can see the serial #

 

Jon

[heffe2001]

Unraid detects the drives correctly it looks like, just looks like it looses it somewhere in there..

 

 

[nojstevens]

Are you running 5beta2?

 

Jon

[heffe2001]

I've tried both, 5b2, and the latest release (just to make sure it wasn't something to do with the beta).  Same results.  I can't use the supermicro firmware as it's for a different revision of the chip (b3 as opposed to my b2), but it works perfectly in esxi.  I guess I could always just use raw disks instead of controller passthrough (I'm only going to be running about 5 drives total on this rig), I'd just like to see temps and have spindown at some point.  I'm wondering if the problem isn't so much in the controller/firmware, but in the linux drivers handling of it.

 

[nojstevens]

What's confusing me is that unRAID can see the serial number in the assign screen but not the array screen. That makes me wonder if the issue isn't related to a unRAID bug? Unless it gets the data from two different places. On Raw Device mode you won't get serials or temps either i don't think. Are there any options in your BIOS to activate S.M.A.R.T. on your controller?

 

Jon

[Joe L.]

What's confusing me is that unRAID can see the serial number in the assign screen but not the array screen. That makes me wonder if the issue isn't related to a unRAID bug? Unless it gets the data from two different places. On Raw Device mode you won't get serials or temps either i don't think. Are there any options in your BIOS to activate S.M.A.R.T. on your controller?

 

Jon

I'd report it to lime-tech.  sounds like a bug to me too.

[heffe2001]

I posted it in the 5.0b2 thread, and emailed support, so we'll see what happens.  If I go to the console of the unraid box, I can use smartctl to get all the correct data from the drives, so it will report it, just for some reason Unraid doesn't read it correctly.  Hopefully it can get straightened out, but now I'm going to see if I can somehow pass through the onboard controller and see if IT works through esxi (it works perfectly through non-vm unraid).

 

[heffe2001]

Looks like my problem is a known thing, and it slated to be fixed at a later date.

 

It looks like the system keeps the drive maps the same between boots, as long as I don't move any drives in the chain.  If they stay the same, it works except for the temp/spindown.

 

[nojstevens]

Heffe,

 

thats good news i guess, but i don't fully understand it, is there a thread that discusses this, i want to avoid it if i can

 

Jon

[heffe2001]

Basically, as long as I don't move a drive to another spot on the controller, unRaid will put them where they are supposed to go in the lineup.  I'm going to order a few more parts, but it'll be a few weeks before I will really get a chance to do much with it (going out of town on business for the next 2 or 3 weeks), so there won't be much as far as updates for a bit from me..

 

 

[tranm5]

I dit some tests this week-end with my p5q-vm do.

 

Using a PCI controller (Promise 300 TX4) passed through the UnRaid VM, I got the temperatures and the drives spin down. Parity Sync run at 30~36MB/s. So its fully fonctionnal but a bit slow...

 

 

Unfortunately I was not yet able to use a controller on the PCIe x16 slot. I tried 2 controller (Promise TX8660 and an adaptec, but I do not remeber the model). With both controller I can't see any drive in the BIOS...