Anyone run unRAID under Xen or KVM?

[josetann]

I currently run unRAID on a stock install of Slackware.  Just finished upgrading from 4.3 to 4.5.4 (which involved upgrading Slackware from 12.0 to 13.0).  After many hours of getting everything to work again, I decided that virtual servers would probably be much easier.  Before I try to reinvent the wheel, has anyone here run unRAID as a Xen or KVM guest?  I'd like to just give the virtual server direct access to the hard drives unRAID would use.  I see I can give Xen guests full access to pci devices, which I would assume means I could give Xen control of the hard drive controllers.  Might work, but I'd need to make sure the boot disk is on its own separate controller.  If I could just give access to /dev/sda without a huge performance hit, I think that'd be best (plus it'd make it easier to swap motherboards down the road if needed).

[WeeboTech]

How would xen hand the USB Key for licensing?

 

I've often thought about using XEN, either having XEN in unRAID so you can run XEN guests or the other way around.

At least if unRAID supported XEN guests, then we could load up a slackware dev environment with ease.

[Ford Prefect]

I currently run unRAID on a stock install of Slackware.  Just finished upgrading from 4.3 to 4.5.4 (which involved upgrading Slackware from 12.0 to 13.0).  After many hours of getting everything to work again, I decided that virtual servers would probably be much easier.  Before I try to reinvent the wheel, has anyone here run unRAID as a Xen or KVM guest?  I'd like to just give the virtual server direct access to the hard drives unRAID would use.  I see I can give Xen guests full access to pci devices, which I would assume means I could give Xen control of the hard drive controllers.  Might work, but I'd need to make sure the boot disk is on its own separate controller.  If I could just give access to /dev/sda without a huge performance hit, I think that'd be best (plus it'd make it easier to swap motherboards down the road if needed).

 

...not as far as into production (with license)....I tested it with an unraid ISO instead of USB

...I'd also prefer KVM instead of XEN...USB access should work, PCI passtrough is also possible

in newest KVM (but hardware needs to support it; intel vt-d extension, i think) and raw access

to host devices is working out of the box.

But you'd suffer from a performance degree, even if you inject virtio-block and -net drivers.

I am currently running openfiler as a guest for my NAS. Average performance is about 60%

than that of a native instance...your mileage may vary.

I strongly recommend to give a look into ProxMox-VE.

It is KVM on a small debian host with Web-based provisioning....rock stable and easy as 1-2-3 for KVM and openvz.

[josetann]

How would xen hand the USB Key for licensing?

 

I've often thought about using XEN, either having XEN in unRAID so you can run XEN guests or the other way around.

At least if unRAID supported XEN guests, then we could load up a slackware dev environment with ease.

 

For now, let's assume we're going to run unRAID in Basic mode.  If I could get it to work the way I want, then I could worry about getting it to recognize my usb key.  Even if it couldn't, others might still find it useful (you could run multiple instances of unRAID, each with two data drives and one parity drive).

 

I'm trying to go the opposite way of what you're thinking about.  Sure, it'd be nice if unRAID supported Xen guests, but I'd rather go the other way around.  Only install security updates on the main OS, it'll remain relatively untouched.  Upgrade Xen guests as often as wanted.  For instance, I could make a backup of my unRAID guest, install a new version, if it works then great!  If not, then just start up the old version.  Upgrading unRAID won't affect anything else on the server (such as asterisk), and vice versa.

[josetann]

 

...not as far as into production (with license)....I tested it with an unraid ISO instead of USB

...I'd also prefer KVM instead of XEN...USB access should work, PCI passtrough is also possible

in newest KVM (but hardware needs to support it; intel vt-d extension, i think) and raw access

to host devices is working out of the box.

But you'd suffer from a performance degree, even if you inject virtio-block and -net drivers.

I am currently running openfiler as a guest for my NAS. Average performance is about 60%

than that of a native instance...your mileage may vary.

I strongly recommend to give a look into ProxMox-VE.

It is KVM on a small debian host with Web-based provisioning....rock stable and easy as 1-2-3 for KVM and openvz.

 

I was seriously considering ProxMox VE.  I'm still debating on whether I should go with Xen or KVM.  I'm leaning towards Xen at the moment, but KVM plays nicer with OpenVZ (I know, there are kernels that support both Xen and OpenVZ, but it's more of a kludge).  I do like the idea of the lower overhead OpenVZ servers would require, but the only thing I can think of I'd use it for is Asterisk, which would need DAHDI support compiled into the host kernel (I could just use Xen/KVM and use a separate kernel, and leave the host kernel alone).

[josetann]

Spent all day with KVM.  Here's what I learned:

 

You can use the kernel and initrd outside of the guest OS.  Upgrading unRAID would be easy; you'd stop the guest OS, replace the bzimage and bzroot on the host OS, then restart the guest.  Voila!

 

Performance didn't suffer that much.  Read speeds were in the low 70s (MB/s) natively, and in the low 60s under KVM.

 

You do need another drive for the guest OS to mount, and has to be labeled "UNRAID".  I didn't get far enough to worry about exposing a usb drive to the guest OS due to issues I ran into.

 

Issue #1.  KVM would recognize hda through hdd.  Trying to specify a drive as hde gave an error message.  Might get further if you combined with other drive types (i.e. scsi).

 

Issue #2.  hdparm could not spin down the drive or get any information from it.   I didn't expect it to, but it would have been nice.

 

Still trying to get scsi to work, but I'll probably give up and try xen later.  I should be able to assign entire disk controllers to a xen guest, that should fix the problems.  Xen doesn't require VT-D to do so, KVM does (VT-D is newer than VT-X).

[josetann]

KVM is supposed to have an option to set the drive's serial number.  I can't get it to work.  I bit the bullet and did a restore (or rather, an initconfig).  Parity check is taking much longer than usual, about ten times as long.  Not sure why, could be because I'm using scsi instead of ide emulation.

 

If I could figure out how to convince unRAID that parity is valid, this would almost be a workable solution.  Waiting over 33 hours for it to rebuild parity is crazy.

[Ford Prefect]

KVM is supposed to have an option to set the drive's serial number.  I can't get it to work.  I bit the bullet and did a restore (or rather, an initconfig).  Parity check is taking much longer than usual, about ten times as long.  Not sure why, could be because I'm using scsi instead of ide emulation.

 

If I could figure out how to convince unRAID that parity is valid, this would almost be a workable solution.  Waiting over 33 hours for it to rebuild parity is crazy.

 

Don't know about setting the serial...what distro are you running your kvm host on

and which kernel and kvm/qemu version are you using?

I found that some features depend heavily on the right combination of the three.

 

Did you try and build a custom kernel with virtio-net and -block drivers for the unRAID guest?

Performance should increase a lot compared to the standard emulation...

[josetann]

Don't know about setting the serial...what distro are you running your kvm host on

and which kernel and kvm/qemu version are you using?

I found that some features depend heavily on the right combination of the three.

 

Did you try and build a custom kernel with virtio-net and -block drivers for the unRAID guest?

Performance should increase a lot compared to the standard emulation...

 

Using Proxmox VE.  Posted about it on their forums, they decided it was a bug in kvm.  I found an old bug report that said it had been fixed.  The version of kvm I'm using should have the patch.  I guess it depends on whether Proxmox VE got kvm from debian (the distro they're built on top of) or straight from the source.

 

Yup, have virtio compiled in.  The OS can see /dev/vdX, but the unRAID gui cannot.  I have to use SCSI emulation (I could use IDE instead, but I'm already using four drives, and want to add more in the future).  I put in a request for /dev/vdX support (at least for emhttp to recognize the drive exists, I can deal with compiling support into the kernel), we'll see.

 

Part of the reason I thought it was sooooooo slow, is because it was recomputing parity, instead of just checking it.  Checking parity ran much faster (still slower than natively though).  Single disk reads are still fast, 60MB/s vs 70MB/s native.  unRAID can't spin the drives down, but I can set the timeout on the host OS ("hdparm -S242 /dev/sda" will set the timeout to one hour).

 

If you build a system with vt-d support, you can simply attach entire drive controllers (and other hardware devices) directly to the guest.  Heck, you could probably boot a stock unRAID setup directly off the usb drive.  Has to be supported by the cpu and the motherboard.  If you bought a high-end Intel system in the past couple years, or a mid-range system in the past year, you might have it.  If you bought an AMD system, you're probably out of luck (they do exist though).

 

Anyways, I'm fairly happy with the current setup.  Now I can mess with unRAID without bringing everything else down.  Have two other VMs running (PBX in a Flash and ZoneMinder).  Very cool stuff.

[Ford Prefect]

Using Proxmox VE.  Posted about it on their forums, they decided it was a bug in kvm.  I found an old bug report that said it had been fixed.  The version of kvm I'm using should have the patch.  I guess it depends on whether Proxmox VE got kvm from debian (the distro they're built on top of) or straight from the source.

Are you running their kernels and VE from their testing branch?

I am not sure where they stick to plain lenny and where they put in extra things.

My PVE host runs kernel "2.6.32-2-pve" from testing while one of my stock lenny VM runs "2.6.26-2-486 on i686"

 

Yup, have virtio compiled in.  The OS can see /dev/vdX, but the unRAID gui cannot.  I have to use SCSI emulation (I could use IDE instead, but I'm already using four drives, and want to add more in the future).  I put in a request for /dev/vdX support (at least for emhttp to recognize the drive exists, I can deal with compiling support into the kernel), we'll see.

Yes, I know of that problem...had the same with my VM running Openfiler...the OF-Gui would simply not parse the list of devices and partitions other than ide and scsi.

Under OF a simple change to a script would do the trick...don't know how emhttp is accessing the info...isn't it based on PHP somewhere down the line?

 

Part of the reason I thought it was sooooooo slow, is because it was recomputing parity, instead of just checking it.  Checking parity ran much faster (still slower than natively though).  Single disk reads are still fast, 60MB/s vs 70MB/s native.  unRAID can't spin the drives down, but I can set the timeout on the host OS ("hdparm -S242 /dev/sda" will set the timeout to one hour).

Yes, I have done the same with my OF installation...works OK.

 

If you build a system with vt-d support, you can simply attach entire drive controllers (and other hardware devices) directly to the guest.  Heck, you could probably boot a stock unRAID setup directly off the usb drive.  Has to be supported by the cpu and the motherboard.  If you bought a high-end Intel system in the past couple years, or a mid-range system in the past year, you might have it.  If you bought an AMD system, you're probably out of luck (they do exist though).

Yes, you're right...but trends for home use go towards more green (Atom-based) solutions, I'd think....so here you do not have a single chance to do any of that with them.

Some AMD Opterons support that Feature (special version of their IOMMU) I think, but I have never seen one.

 

Anyways, I'm fairly happy with the current setup.  Now I can mess with unRAID without bringing everything else down.  Have two other VMs running (PBX in a Flash and ZoneMinder).  Very cool stuff.

Keep us posted on how you play along.

I am intending to swap my setup from AMD64 based proxmox host and Openfiler NAS(VM) into an Atom

based native unRaid host and run my additional VMs under virtualbox on it.