UnRAID on VMWare ESXi with Raw Device Mapping

[Ford Prefect]

still my issue is that i can't get an iso or anything on a usb to be seen by ESXi.

 

Even the usb that it is installed on isn't visible when i do an fdisk -l (or -ls?).

I actually never tried to install ESXi on a flash...mine is on a SDD 

I've been under the assumption, that ESXi would reserve some space on the flash as datastore.

What is a vsphere client reporting in the config section under "storage" ?

 

the drive that unraid is on is bigger, but it's busy on another machine pre-clearing the drive that esxi hijacked.

 

if all else fails, i guess i need to get a wd scorpio or something to get moving on this.

yes, a bigger drive or flash should take you further...

[axeman]

still my issue is that i can't get an iso or anything on a usb to be seen by ESXi.

 

Even the usb that it is installed on isn't visible when i do an fdisk -l (or -ls?).

I actually never tried to install ESXi on a flash...mine is on a SDD 

I've been under the assumption, that ESXi would reserve some space on the flash as datastore.

What is a vsphere client reporting in the config section under "storage" ?

 

the drive that unraid is on is bigger, but it's busy on another machine pre-clearing the drive that esxi hijacked.

 

if all else fails, i guess i need to get a wd scorpio or something to get moving on this.

yes, a bigger drive or flash should take you further...

 

i actually installed from a flash, to a flash - go figure. after much hair pulling and buggy bootloaders, i finally got it working going the 4.0 to 4.1 upgrade route.

 

after the pre-clear ends on the other machine, i'm going to take the UR stick and try install ESXi on that. it's bigger, faster and supposedly "better". i'll be the judge of that 

[Johnm]

I have my ESXi booting with a 4GB flashdrive. Like you, i saw no option to build the datastore on the flash.

 

in the end i used crappy old 7200rpm 3.5" for my datastore for now.

 

next rebuild i am going to move it all to an SSD or 7200rpm laptop drive and leave it inside my norco.

 

Im just irritated that my test board has an Intel card that is to new to use. I'll have to try the community OEM driver route I think.

 

Anyone know of a windows tutorial for doing that?

 

I guess i can get a Linux live CD to build the image. I was just trying to avoid that.

 

that or order a $25 Intel card.

[axeman]

I have my ESXi booting with a 4GB flashdrive. Like you, i saw no option to build the datastore on the flash.

 

in the end i used crappy old 7200rpm 3.5" for my datastore for now.

 

next rebuild i am going to move it all to an SSD or 7200rpm laptop drive and leave it inside my norco.

 

Im just irritated that my test board has an Intel card that is to new to use. I'll have to try the community OEM driver route I think.

 

Anyone know of a windows tutorial for doing that?

 

I guess i can get a Linux live CD to build the image. I was just trying to avoid that.

 

that or order a $25 Intel card.

 

i'm going to try again later tonight on the Patriot XT 32GB card. figure force the bios to show it as HD... don't know if it'll help. ESXi 4.1 manual says you can't create datastore on IDE, which is probably what the bios is emulating the USB sticks as.

 

i kick myself for not buying the $25 scorpio blue from Newegg a couple of weeks ago.

 

Get the $25 nic, and call it a day. i got two, ESxi is kind enough to offer to team them, VM traffic, or redundancy. I've left it as just redundant. I don't know about Teaming or ESXi enough yet to monkey with those.

 

While it is just a network card, it's got blazin reviews, 90% at 5 Eggs... don't think there are too many products that get that kinda approval rating!

[axeman]

Fail, Fail and fail.

 

all attempts to use the USB as a datastore for the UnRaid VM are bust.

 

i will breakdown and get something real soon. I have a 1TB WD Black and a 500GB WD MyBook that was turned ext to internal. I'll either get an SSD to replace the black and then use the black in esxi or just grab the crappy 500GB and use that.

 

----

 

Also this whole plop setup is an alternate, right? the procedure oulined in the OP would still technically work? With all the trouble I had on ESXi (my fault) - i just don't want to add another thing to the learning box.

[chetebrown]

I've decided to take the plunge and try consolidating my physical unraid to ESXi after the reports of working spindown in the latest beta with the br10i.

 

I did see some posts of returns from eBay vendors so I was looking for recommendations on where to get one reliable and relatively affordable br10i.

 

 

 

[jortan]

As I understand it, this card ships with some IBM servers - in some cases a different disk controller is required and so the BR10i is surplus to requirements - this is why you'll see a good number of effectively new BR10i's on ebay at a keen price.

 

"returns from eBay vendors" as in the card was faulty?  You could buy elsewhere but you will probably pay more - I don't see the risk of DOA as being much higher on ebay for these cards...

 

ps: Try to get the bracket included, apparently it can be difficult to source by itself as evidenced by the price gouging for the bracket alone on ebay.

[madburg]

Quickly,

 

1) Recommended to run ESXi from a USB stick, installing to a drive is a waste. Your are installing a hypervisor on the USB stick.

 

2) The USB stick cannot be a datastore. So a 4 GB stick is more than enought.

 

3) IF you have drives installed in your chassis when you boot ESXi for the first time it WILL blow away anything you have on them as it wants to create 1 datastore in order to be fully functional (to be clear if you have no RAID it will use the first disk and blow it away. If you have sasy RAID 5 with 10 disks in it it will blow away the 10 disks you have set in the RAID 5), it will create a folder named "locker" and drop and use this area for logs, writes, etc.. Do not try to configure anything until you have 1 datastore set. I always remove all disks upon a new ESXi install. Because the default VMFS block size is 1MB, depending on what you are looking to achieve you may want a 8MB block size. So again I remove all disks boot for the first time. It will complain there is no datastore (if you vClient into it). Shutdown, add the disk or disks you wish to have a datastore on. Boot back up, no it will not automatically blow away any disk and create a datastore on is own. Proceed to create a datastore, select the block size you wish to have for VMFS. Reboot once more, the "locker" folder will be created (you can browse via Datastore browser to see this). Your all set now to passthrought devices, create raw drive mappings, vNetworks, etc...

 

4) ESX4.x automatically passes through usb sticks. Ex. first stick is your ESXi install, second stick is your unRAID install. Once you create you VM and add USB Controller to it (save) go back to edit settings add USB, you will see your unRAID stick as an option to select (this is the auto passthrough part, you do NOT do this is the advance options to passthrough nic, controllers, if you pass throught your USB controller there, you will never boot ESXi again and will have to start all over). Now currently ESXi does not have the ablility for a VM to boot from USB. So even thought you just set up the unRAID stick to the VM you need something to get this VM to boot the stick, this is were PLOP comes in. Mount the PLOP ISO (you copied this to your datastore first right ) mount the iso to your CD/DVD in the VM. Boot and using the menu have it install to the small disk you created (VMDK). Unmount the PLOP ISO. Boot again this time you would be booting PLOP from the disk (VMDK) and now you can set it with preferences (ex. make it only look to boot from USB, 3 sec boot menu, etc...)

Now this VM (that you named unRAID right ) when powered on will boot PLOP from HD, which will boot your unRAID USB key.

 

5) You can now proceed to the advanced options and passthrough any hardware devices (ex. Nic, SATA/SAS controllers) you need that you want to assign to your unRAID VM. Everytime you add or remove devices in the advance option for passthrough you will need to reboot the ESXi host.

[Johnm]

Quickly,

 

1) Recommended to run ESXi from a USB stick, installing to a drive is a waste. Your are installing a hypervisor on the USB stick.

 

This is what i am doing now. I was going to use a 2.5 drive for my datastore since i will have several windows clients.

Can i put the hypervisor and the data store on the same drive, or must they stay separate?

 

These supermicro serverboards are seriously devoid of USB ports 

[gfjardim]

Anyone has unRAID the beta.6d to version to share? I want to test the compatibility with the device mapping settings.

[madburg]

Quickly,

 

1) Recommended to run ESXi from a USB stick, installing to a drive is a waste. Your are installing a hypervisor on the USB stick.

 

This is what i am doing now. I was going to use a 2.5 drive for my datastore since i will have several windows clients.

Can i put the hypervisor and the data store on the same drive, or must they stay separate?

 

These supermicro serverboards are seriously devoid of USB ports  

 

You can have both on one HD drive (not usb) but not recommended. They will be on different partitions (the hypervsior contains more than one partition. All unallocated space can be used as a datastore).

[axeman]

Quickly,

....

 

Hacker!! you been watching me or something? lol this is *exactly* what I painstakingly learned over the last week or two. Part of what you said in 4 and 5 I haven't gotten to yet, but will very very soon.

 

Oh and ty... I was about to say screw the USB hypervisor, and put everything on the soon to be datastore drive.

 

All that being said my goal still is to use the UnRaid protected drives as NFS datastores. Don't know that it will work, or properly. still got a long way to go before I get there.

[madburg]

Quickly,

....

 

Hacker!! you been watching me or something? lol this is *exactly* what I painstakingly learned over the last week or two. Part of what you said in 4 and 5 I haven't gotten to yet, but will very very soon.

 

Oh and ty... I was about to say screw the USB hypervisor, and put everything on the soon to be datastore drive.

 

All that being said my goal still is to use the UnRaid protected drives as NFS datastores. Don't know that it will work, or properly. still got a long way to go before I get there.

 

I work alot with VMWare products at work (I have been worked with &rolled out ESXi since 3.5). This is my first experience having to get a VM to work with passthrough devices though (Plop, etc..). And having my own interesting issues at this point (unRaid is not loading the second LSI controller i have installed. "lsmod" show its there... Same with onboard NIC, "lsmod" shows it is there but wont load it. If I issue it a vNic its fine. My goal is to passthrough all hardware unRAID will use, so nothing seems virtuallized to unRAID).

 

Everyone needs to keep in mind this is VMWare first pass at hardware passthrough and actually designed only for network cards. Their idea based on requests, etc.. was a Guest (VM) that required big bandwidth not virtualized and not shared with others. So by being able to passthrough say a 10GB Network card it meets the purpose. I know in the next release you will be able to allocated network bandwidth slicing, ex.. physical 10G Nic, allocate a 2GB slice to VM#1, all other VM get a 1GB slice. Not sure if passthrough will be expanded on. I do hope so and there are many cool things that can be done. So being able to passthrough other devices is all beta at best. Thats why sharing info on what you got to work helps everyone out.

[axeman]

I might have missed something - but is there a definitive advantage of going the plop route? If i create a vmhd from the flash can i still boot from it ?

 

Just asking since I don't know much and plop would mean adding another layer of complexity to it. Also feel like when I do go to the paid version, the flash would be more "durable" if it's locked away somewhere.

 

Thanks!

[axeman]

stupid stupid stupid. got it!

 

"... /boot will be mounted from your USB stick, so your array config and any changes"

 

plop it is!

[axeman]

okay - plop working, esxi working, unraid working (haven't added drives to array yet)..

 

thanks all... now I have to figure out the Paravirtualized? setup so i can get my drive temps. which are more important to me that i initially thought.

 

the IBM BR10i has two sata ports? so bare minimum for UR, you'd need two of them, right?

 

One other question to the smarter than me people here ... if I pull out the ESXi usb stick, and let the unraid usb stick run, will everything work just dandy?

 

Oh another last question - my UR stick is at 4.7, do i need to bring it to the 5 beta for this to work properly?

[madburg]

okay - plop working, esxi working, unraid working (haven't added drives to array yet)..

 

thanks all... now I have to figure out the Paravirtualized? setup so i can get my drive temps. which are more important to me that i initially thought.

 

the IBM BR10i has two sata ports? so bare minimum for UR, you'd need two of them, right?

 

One other question to the smarter than me people here ... if I pull out the ESXi usb stick, and let the unraid usb stick run, will everything work just dandy?

 

Oh another last question - my UR stick is at 4.7, do i need to bring it to the 5 beta for this to work properly?

Why Paravirtualized, when you can just passthrough the IBM BR10i cards and assign them to the unRAID VM? (this is assuming u are using one or more hard drives for the ESXi datastore that is NOT on a IBM BR10i(s) ex. on board SATA port)

DO NOT remove the ESXi usb stick once its booted, big no no.

You will need at least unRAID 5.0Beta6a for the IBM BR10i's (you will get MPTSAS driver with that version to load the card/s and Drive temps for any hard drives on those cards), if you emailed Tom back when he offered unRAID 5.0Beta6d for testing (private beta) you would also get spin up/down support (native).

[axeman]

okay - plop working, esxi working, unraid working (haven't added drives to array yet)..

 

thanks all... now I have to figure out the Paravirtualized? setup so i can get my drive temps. which are more important to me that i initially thought.

 

the IBM BR10i has two sata ports? so bare minimum for UR, you'd need two of them, right?

 

One other question to the smarter than me people here ... if I pull out the ESXi usb stick, and let the unraid usb stick run, will everything work just dandy?

 

Oh another last question - my UR stick is at 4.7, do i need to bring it to the 5 beta for this to work properly?

Why Paravirtualized, when you can just passthrough the IBM BR10i cards and assign them to the unRAID VM? (this is assuming u are using one or more hard drives for the ESXi datastore that is NOT on a IBM BR10i(s) ex. on board SATA port)

DO NOT remove the ESXi usb stick once its booted, big no no.

You will need at least unRAID 5.0Beta6a for the IBM BR10i's (you will get MPTSAS driver with that version to load the card/s and Drive temps for any hard drives on those cards), if you emailed Tom back when he offered unRAID 5.0Beta6d for testing (private beta) you would also get spin up/down support (native).

 

Thanks!

 

oh - because i don't have the IB BR10i. running of the motherboard headers at the moment. when i started this, drive temps were a "nice" to have feature. after running my pre-clears, i NEED it.

 

as for removing the USB, i meant a graceful, powered down machine... supposing i screw esxi up (again)... can i just boot to UnRaid and still see my shares?

 

i'll email Tom - i'm a non-paying customer, so don't know if he'll reply back. I would LOVE it if I can get the drive temps/spin up/down working off the motherboard headers.

 

going to run a network speed test to see how fast I can write to the array.

[madburg]

You need to slow down a bit, and breath. I am having a hard time figuring some of your abbreviations and thoughs in your question.

 

Yes u can shutdown your box, remove the ESXi USB key and just boot the unRaid USB key and it will see everything just the same, disk assignments and all. It's a beautiful thing. Give it a quick try to see for yourself. U may have to change the boot order of the USB key in some MB bios when u do this though. You will have HD temps with 5.0beta6b for sure. Let Tom work his majic to get us 5.0beta7, so hold tight. As there are so other bugs in 6d. U won't die because u can spin down 3 drive (with the free version for now)

[axeman]

You need to slow down a bit, and breath. I am having a hard time figuring some of your abbreviations and thoughs in your question.

 

Yes u can shutdown your box, remove the ESXi USB key and just boot the unRaid USB key and it will see everything just the same, disk assignments and all. It's a beautiful thing. Give it a quick try to see for yourself. U may have to change the boot order of the USB key in some MB bios when u do this though. You will have HD temps with 5.0beta6b for sure. Let Tom work his majic to get us 5.0beta7, so hold tight. As there are so other bugs in 6d. U won't die because u can spin down 3 drive (with the free version for now)

 

Thanks. I'm going to stick with 4.7, at least for my testing purposes. my production machine (WHS) is full. At some point I need to move away from it. Are people considering this a stable setup? I know OP said not to be silly and put valuable data on the setup...

 

I was re-reading what I wrote, and I think you answered most of my questions - sorry if they weren't all clear. I am a little giddy that I got this mostly working. read/write from another desktop to a share on unraid was really fast. Using Lan Speed Test, i got 86MBs write and 68MBs read (I thought these should be flipped... but not complaining). That was without a parity drive assigned.

 

Waiting on a parity calculation to test it again.

 

this is good stuff

[axeman]

Okay - so it's the noob again, about that IBM BR10i... (sorry I know it's a bit OT) ... but from what I understand, it can support 8 devices? so I would get a breakout cable like this one: http://www.newegg.com/Product/Product.aspx?Item=N82E16816151080 ?

 

which would allow me to hook-up my 3x WD20EARS drives?

[BRiT]

You will need at least unRAID 5.0Beta6a for the IBM BR10i's (you will get MPTSAS driver with that version to load the card/s and Drive temps for any hard drives on those cards), if you emailed Tom back when he offered unRAID 5.0Beta6d for testing (private beta) you would also get spin up/down support (native).

 

No need to email Tom/Limetech. Use unRAID 5.0beta7. It has all the spinup/spindown/drive-temps for HBAs including the IBM BR10i, native.

[axeman]

You will need at least unRAID 5.0Beta6a for the IBM BR10i's (you will get MPTSAS driver with that version to load the card/s and Drive temps for any hard drives on those cards), if you emailed Tom back when he offered unRAID 5.0Beta6d for testing (private beta) you would also get spin up/down support (native).

 

No need to email Tom/Limetech. Use unRAID 5.0beta7. It has all the spinup/spindown/drive-temps for HBAs including the IBM BR10i, native.

 

Awesome! Thanks!

 

Is there a way, via the magic of Linux or ESXi, that I can take ALL the UnRaid user shares, and export them to another VM so that they appear as a physical HD to that machine (as opposed to a mapped network drive)?

 

 

[madburg]

You will need at least unRAID 5.0Beta6a for the IBM BR10i's (you will get MPTSAS driver with that version to load the card/s and Drive temps for any hard drives on those cards), if you emailed Tom back when he offered unRAID 5.0Beta6d for testing (private beta) you would also get spin up/down support (native).

 

No need to email Tom/Limetech. Use unRAID 5.0beta7. It has all the spinup/spindown/drive-temps for HBAs including the IBM BR10i, native.

Thats what i get for not having a notify setup on the announcement page  , thanks BRiT cant wait to try beta7.

[gfjardim]

I've tested here and unRAID v5b7 now support spinup/spindown the drives with Mapped Raw LUN with either LSI Logic SAS (mptsas driver) and Paravirtual SCSI (using recompiled kernel with PVSCSI driver) controllers. Temperatures still a non go, but I can read them with smartctl, so it shouldn't be hard to fix it.

 

Why do I insist using Mapped Raw LUN and not passthrough the controller? Because VT-d and IOMMU capable hardware are more expensive and hard to find, while MRL only requires a compatible disk controller, e.g. LSI 1068E/LSI 2008. No need to have Xeon processor and ECC RAM. Any computer with a cheap PCI Intel card should do the trick.

 

Did Tom express any interest in support unRAID for ESXi? Maybe anyone can talk to him asking the temperature bug to be fixed.