UnRAID on VMWare ESXi with Raw Device Mapping

[SilntBob]

I have no idea what performance would be like running the VMDKs from the UnRAID array, but with the UnRAID write speed and IO op overhead, I wouldn't recommend running the primary virtual disk from the array.  You are going to need SOME sort of drive to boot ESXi from and have an ESXi datastore, etc.

 

Here are the options if you go ESXi

1 drive= ESXi + datastore for VMDK.  If you want to use the "extra" space on this drive, you *could* create a virtual drive on a portion of this space and use that virtual drive as a disk in your UnRAID array....you could also use the virtual drive for the UnRAID cache drive.

 

Presently, I'm currently running a 1 drive Datastore setup, but I'm going to be changing that in the near future to:

 

Using a perc5i raid controller for the ESXi Datastore...  I'm going to set up a HW RAID array using some older disks.  The HW RAID will support the ESXi installation as well as the datastores for my VMs....this keeps my on-board SATA ports free for pass-through to the UnRAID VM.

 

 

Additionally, you aren't going to be able to run the VMs off of the UnRAID array, because UnRAID doesn't provide iSCSI pass-through to present the storage array to ESXi.....you are going to need at least one drive to host the ESXi installation + datastore.

 

As far as Carbonite goes, Carbonite is a decent program and I'm sure you got a good deal on it, however, it has too many limitations that prevent it from being nearly as well suited for this use as CrashPlan.  I used to use Carbonite and I've been 100x happier since I switched....the last time I checked, Carbonite doesn't have a linux client and it doesn't let you back-up a network location to the cloud....it isn't going to work for cloud-syncing an UnRAID server, Crashplan does.

 

If you are interested in moving forward and have a server you can play around with, first try installing ESXi on it, just to get a feel for it.....it is different with a slight learning curve.

Then, install UnRAID on a USB stick and play around with *that* on the server.....THEN try to build the UnRAID VM to run inside of ESXi.

 

 

Now that I'm getting the kinks worked out of the system I'm really happy with it, and I pleased to be able to get everything consolidated to run on a single server for the house.

[SilntBob]

You could export an unRAID share via NFS and use that as a datastore but as has already been pointed out, performance wouldn't be so great not to mention I didn't find it all that reliable (at least with unRAID running as a VM).

 

Can you?  I know you can use iSCSI as a datastore destination, but I didn't know you could do it with NFS....

[dyrewolfe]

You could export an unRAID share via NFS and use that as a datastore but as has already been pointed out, performance wouldn't be so great not to mention I didn't find it all that reliable (at least with unRAID running as a VM).

 

Can you?  I know you can use iSCSI as a datastore destination, but I didn't know you could do it with NFS....

 

With NFS, yes. Samba/CIFS, no. Of course iSCSI and FC are also supported.

[axeman]

silntbob - i do have the hardware, and will be toying with this stuff in the coming weeks. as of right now, i either have an overpowered NAS or a rightsized all you can do vm megabox.

 

i like the perc card - but i think i paid less than that for all my other parts combined. would be wayyyy outta budget.

 

my current machine transfers at 24MBytes read and 21MBytes write. I was mistaken, it *is* on a gigabit card. although technically it is full duplex, in reality, it isn't. When HTPC rips a movie and shuttles it over to the WHS , I cannot simultaneously stream something else from the WHS. chalk that up to aged hardware.

 

For now, I have gotten some workarounds - like I'll rip at off hours, etc. IF this UR setup can at least meet that transfer spec, I'd be semi-satisfied. IIRC somewhere someone on here said that URs rates are along the 50Mbyte mark?

 

My ESXi and/or UR installation would be stored on a Patriot XT flash drive. it's got 32GB on it, so would have been nice to use it for both of those.

 

i'm guessing from what dyrewolfe said, and what you've said - this kinda thing isn't going to fly, even with the i7. I had high expectations

 

 

[dyrewolfe]

For now, I have gotten some workarounds - like I'll rip at off hours, etc. IF this UR setup can at least meet that transfer spec, I'd be semi-satisfied. IIRC somewhere someone on here said that URs rates are along the 50Mbyte mark?

 

My ESXi and/or UR installation would be stored on a Patriot XT flash drive. it's got 32GB on it, so would have been nice to use it for both of those.

 

 

On my setup, reading/writing directly to the protected array yields roughly 35-40MB/s. With a cache drive (I'm using a 2TB Hitachi as a warm spare/cache drive), you can easily double those figures.

 

Although I'm sure it's trivial to dual-boot ESXi and unRAID from the same key, if you wanted to use unRAID as a VM you'll have to give it exclusive access to the USB controller that they unRAID key resides on. I'm fairly certain ESXi doesn't allow the primary install device to be shared.

 

I have ESXi installed on a 4GB key while unRAID resides on a separate 2GB key.

[axeman]

getting another flash isn't too much of a big deal. I'm guessing give the faster one to UR, and get a not so great one for UR ?

 

so basically if i can get 1/2 your speed, i'd be happy.

 

I saw somewhere in the ESXi manual that you can create on large VHD or have it split files (which the manual said was not reccomended for performance reasons)...

 

so if i did fail to heed all advice, and try this, i'd hit two performance warnings.

 

:'(

[dyrewolfe]

I don't think it would matter a whole lot. Both will load to RAM and I doubt you'll want to reboot much once you have everything set up. In fact, I would almost suggest giving the faster key to ESXi. Passthrough USB takes a performance hit so the unRAID key is rather slow to boot (roughly USB 1.1 speeds). Not a real big deal since I usually don't reboot it too often and once it's loaded, there is very little r/w from the key itself. There are other things you can do such as clone the unRAID key to a VMDK and boot from that instead, but you'll still need the key for licensing purposes. I think that method is outlined on the first page of this thread.

 

Of course, YMMV so I would experiment till you are satisified with the performance.

[axeman]

I don't think it would matter a whole lot. Both will load to RAM and I doubt you'll want to reboot much once you have everything set up. In fact, I would almost suggest giving the faster key to ESXi. Passthrough USB takes a performance hit so the unRAID key is rather slow to boot (roughly USB 1.1 speeds). Not a real big deal since I usually don't reboot it too often and once it's loaded, there is very little r/w from the key itself. There are other things you can do such as clone the unRAID key to a VMDK and boot from that instead, but you'll still need the key for licensing purposes. I think that method is outlined on the first page of this thread.

 

Of course, YMMV so I would experiment till you are satisified with the performance.

 

thanks - i'm going to experiment, since i have the gear and it's a "fresh" build. the other machine runs fine, etc. i just hope i am not setting myself up for failure or data corruption that can't be replicated due to the layers of configs. at this point, it's probably unlikely that a failed disk can be moved to another physical machine for diagnosis.

[SilntBob]

i like the perc card - but i think i paid less than that for all my other parts combined. would be wayyyy outta budget.

 

I think I paid about $100 or $125 for mine off ebay, with cables.....either perc cards are cheaper than you think, or you got a hell of a deal on your other components....those cards are so common that at used prices they should certainly be within budget.

 

I'm still working on my performance and I don't yet have a cache drive configured (it is temporarily being used for something else) but read performance has been just fine, write performance tends to be a little slow, but the rest of the system is cooking just fine.

I've got another VM that is acting as my SageTV/Playon/Twonky transcoder box...Initially I have a dedicated drive for that VM to use for recording to, but once I get the cache drive implemented I'm going to consider recording TV directly to the array

[axeman]

FYI - under ESXi for VMs backup I start using script from http://communities.vmware.com/docs/DOC-8760, which pretty smart and skip physical RDMs (if someone want to backup unRAID VM). It even support gzip compression for small VMs

 

I'm getting set up with this script now also.  My ESXi server and unRAID server are separate however.  Currently I'm having ESXi create NFS datastore that points to one of my unRAID shares and that is working.  

 

But I'm not sure how to get the email part working.  The ghettoVCD script doesn't take parameters for SMTP authentication and it's using nc internally and I don't know if it can handle auth either.

Any suggestions or help would sure be appreciated.

 

like this! :-) but want esxi and ur to be same box! what kinda speeds are you getting on your setup queeg?

[axeman]

i like the perc card - but i think i paid less than that for all my other parts combined. would be wayyyy outta budget.

 

I think I paid about $100 or $125 for mine off ebay, with cables.....either perc cards are cheaper than you think, or you got a hell of a deal on your other components....those cards are so common that at used prices they should certainly be within budget.

 

I'm still working on my performance and I don't yet have a cache drive configured (it is temporarily being used for something else) but read performance has been just fine, write performance tends to be a little slow, but the rest of the system is cooking just fine.

I've got another VM that is acting as my SageTV/Playon/Twonky transcoder box...Initially I have a dedicated drive for that VM to use for recording to, but once I get the cache drive implemented I'm going to consider recording TV directly to the array

 

i thought the perc cards are like $600+ havent' looked into them in a while.

 

lotsa things to think about here.

[SilntBob]

i like the perc card - but i think i paid less than that for all my other parts combined. would be wayyyy outta budget.

 

I think I paid about $100 or $125 for mine off ebay, with cables.....either perc cards are cheaper than you think, or you got a hell of a deal on your other components....those cards are so common that at used prices they should certainly be within budget.

 

I'm still working on my performance and I don't yet have a cache drive configured (it is temporarily being used for something else) but read performance has been just fine, write performance tends to be a little slow, but the rest of the system is cooking just fine.

I've got another VM that is acting as my SageTV/Playon/Twonky transcoder box...Initially I have a dedicated drive for that VM to use for recording to, but once I get the cache drive implemented I'm going to consider recording TV directly to the array

 

i thought the perc cards are like $600+ havent' looked into them in a while.

 

lotsa things to think about here.

New price, probably......off-warranty ebay price is much less, which is nice, because they are solid cards.

[dyrewolfe]

thanks - i'm going to experiment, since i have the gear and it's a "fresh" build. the other machine runs fine, etc. i just hope i am not setting myself up for failure or data corruption that can't be replicated due to the layers of configs. at this point, it's probably unlikely that a failed disk can be moved to another physical machine for diagnosis.

 

Actually, due to the nature of unRAID, you can take a data disk from the array and mount it in another machine if need be. The filesystem used is ReiserFS so most any linux distro (or I suppose Windows if you have the right driver) can mount/manipulate it.

[axeman]

thanks - i'm going to experiment, since i have the gear and it's a "fresh" build. the other machine runs fine, etc. i just hope i am not setting myself up for failure or data corruption that can't be replicated due to the layers of configs. at this point, it's probably unlikely that a failed disk can be moved to another physical machine for diagnosis.

 

Actually, due to the nature of unRAID, you can take a data disk from the array and mount it in another machine if need be. The filesystem used is ReiserFS so most any linux distro (or I suppose Windows if you have the right driver) can mount/manipulate it.

 

yeah that i had read... but if i do something like UR, and an NFS datastore that would host a Windows guest ... figure too many layers of filth to dig thru if something goes south.

 

i *do* have two machines here, comprable hardware. suppose I can make one a esxi and the other a UR - similar to what queeg has set up. just a pita to do that, since the other machine is my real, live, working htpc.

[dyrewolfe]

I'd still separate the datastore from the unRAID array if at all possible. Use the unRAID array as a place to backup your VMs.

[SilntBob]

I'd still separate the datastore from the unRAID array if at all possible. Use the unRAID array as a place to backup your VMs.

I'll definitely second that idea.....once I get my cache drive in place, I'll try mounting a VM on UnRaid via an NFS mount, just for giggles to see what the performance is like, but I'm not planning on that for "production" purposes.

[dyrewolfe]

exporting a hidden directory on the cache drive may very well be viable. I'll have a go at that also when I get a few minutes. It might make a nice, occasional use extra datastore.

[SilntBob]

exporting a hidden directory on the cache drive may very well be viable. I'll have a go at that also when I get a few minutes. It might make a nice, occasional use extra datastore.

but if you are using the cache drive, you don't get any parity protection, right?  At that point, what is the point?

[dyrewolfe]

exporting a hidden directory on the cache drive may very well be viable. I'll have a go at that also when I get a few minutes. It might make a nice, occasional use extra datastore.

but if you are using the cache drive, you don't get any parity protection, right?  At that point, what is the point?

 

no you don't so you need to have a decent backup plan in place. Should be easy enough since you just have to copy to the array.

[axeman]

exporting a hidden directory on the cache drive may very well be viable. I'll have a go at that also when I get a few minutes. It might make a nice, occasional use extra datastore.

but if you are using the cache drive, you don't get any parity protection, right?  At that point, what is the point?

 

no you don't so you need to have a decent backup plan in place. Should be easy enough since you just have to copy to the array.

 

Oh - i thought that you can use a cache drive and a parity drive. still waiting to hear back from queeg. this weekend i will start benching this. time is limited so, it might be 50% done, then pick up again next weekend.

 

somewhere deep down inside in places i dont' talk about at parties, i am hoping beyond hope that I can have a beast of a protected VM hosting media sharing behemoth. at this point to me this is like performing brain surgery while flying a rocketship. 

[dyrewolfe]

Oh - i thought that you can use a cache drive and a parity drive. still waiting to hear back from queeg. this weekend i will start benching this. time is limited so, it might be 50% done, then pick up again next weekend.

 

Yes, you can use a cache drive but it is outside of the parity protected array. That's how you get the speed benefits. Any new data you add to the share will actually be written to the cache drive and then later moved to the protected array. Anything in a hidden directory on the cache drive will not be moved (and thus not be protected).

[axeman]

Oh - i thought that you can use a cache drive and a parity drive. still waiting to hear back from queeg. this weekend i will start benching this. time is limited so, it might be 50% done, then pick up again next weekend.

 

Yes, you can use a cache drive but it is outside of the parity protected array. That's how you get the speed benefits. Any new data you add to the share will actually be written to the cache drive and then later moved to the protected array. Anything in a hidden directory on the cache drive will not be moved (and thus not be protected).

 

ah - makes sense - and if the cache drive fails while the data is in-flight to the array, your toast...

[dyrewolfe]

Yes but anything on the cache (besides the hidden directories of course) is moved every 24 hours by default. Depending on your usage patterns, that could be a couple of gigabytes to a couple of terabytes. That's why it's important to do manual backups of any data that is permanently kept on the cache drive.

[stradle]

Hi,

I have finally managed to install ESXi 4.1 on to a USB drive (needed to use the NIC drivers modified ISO as my NIC is Realtech as mentioned in this link http://engwar.com/post/392)

When I go to step 2 of the guide on the first page, I hit some stumbling blocks.

 

My hardware setup is:

 

Antec Nine Hundred Two Case

Intel Core i3 540 3.06G s1156

Asus P7H55-M/USB3 - Motherboard

CAIR 650W CMPSU-650TXUK

CORSAIR 4GB XMS3 PC3-10666MHz

5x 2Tb HDD (3x Western Digital, 2x Samsung)

 

# Identify your disks
fdisk -l | grep '^Disk'
...
Disk /dev/disks/t10.ATA_____ST3160812AS_________________________________________5LS3P8SB
Disk /dev/disks/t10.ATA_____WDC_WD1600JS2D75NCB3__________________________WD2DWCANM7450414
Disk /dev/disks/t10.ATA_____WDC_WD1600JS2D75NCB3__________________________WD2DWCANM7453963
...

mkdir /vmfs/volumes/datastore1/UnRAID
cd /vmfs/volumes/datastore1/UnRAID
vmkfstools -a lsilogic -z /vmfs/devices/disks/t10.ATA_____ST3160812AS_________________________________________5LS3P8SB   mydisk1.vmdk
vmkfstools -a lsilogic -z /vmfs/devices/disks/t10.ATA_____WDC_WD1600JS2D75NCB3__________________________WD2DWCANM7450414   mydisk2.vmdk
vmkfstools -a lsilogic -z /vmfs/devices/disks/t10.ATA_____WDC_WD1600JS2D75NCB3__________________________WD2DWCANM7453963   mydisk3.vmdk
# NOTE: you we're using /vmfs/devices/disks here ... not /dev as above!


--------------------------------------------------------------------------------

 

 

fdisk -l | grep '^Disk'

This is only showing up USB drives, none of my 3 bay SATA backplane drives are showing up which I use in UnRaid

http://www.scan.co.uk/products/jou-jye-st-3051ss-525-(3-bay)-backplane-for-5x-35-sas-sata-hdd

 

Anyone know why this is?

 

The next issues I have is that whenever I try and make a directory (mkdir /vmfs/volumes/datastore1/UnRAID)

It is also stopping me from creating this.

 

login as: root
[email protected]'s password:
You have activated Tech Support Mode.
The time and date of this activation have been sent to the system logs.

VMware offers supported, powerful system administration tools.  Please
see www.vmware.com/go/sysadmintools for details.

Tech Support Mode may be disabled by an administrative user.
Please consult the ESXi Configuration Guide for additional
important information.

~ # fdisk -l | grep '^Disk'
Disk /dev/disks/mpx.vmhba33:C0:T0:L0: 3170 MB, 3170893824 bytes
~ # mkdir /vmfs/volumes/datastore1/UnRAID
mkdir: cannot create directory '/vmfs/volumes/datastore1/UnRAID': No such file o                      r directory
~ #

 

it seems everyone else is having trouble in this thread after getting the initial setup complete without any hitches.

 

Any help on these topics would be appreciated.

 

 

[ChaOConnor]

Stradle, looks like that backplane isn't supported by ESXi.  When you're in vSphere, under Configuration -> Storage -> Devices, do you see your hard drives?  I'm going to assume no, based of of your fdisk -l report.

 

Does your motherboard support VT-d?  Can you pass through the backplanes direct to the unRaid VM?

 

I'm actually in the process of writing up my guide for getting unRaid to work in ESXi, which varies slightly from this guide as mine will focus more on passed through hardware.  I should have it published in a day or two, maybe it will help.