Calling All Xen Users...Feedback Needed...

[jonp]

This message is for anyone that is currently using a Xen-based virtual machine in the unRAID 6 beta.

 

I'm writing to you all tonight to ask for your input on a matter concerning the future of Xen in unRAID.  With the latest release of beta 15 out to the masses, we had some time to experiment with Xen 4.5, the latest release from the Xen team.  From this experimentation, we can say that while Xen has made some strides in a few areas, the general user experience hasn't changed much.  And while we were hopeful that libvirt integration would get to feature parity with Xen's tools, it hasn't.  And this brings us to why I'm here asking for your feedback.

 

With Docker being capable of supporting any Linux application we want without requiring special hardware (VT-x/AMD-v) and QEMU/KVM providing better support for hardware-assisted virtualization (Windows, PCI device assignment, etc.), what does Xen offer that these other two solutions do not?

 

Please let us know your existing use case for Xen and how much effort you feel it would take you to convert that environment to using Docker Containers and/or Kernel-based Virtual Machines.  Thank you.

 

[sparklyballs]

i feel a schism coming, lol.

 

my first foray's into VM's were with xen and I liked it, i like xencenter in particular, but i'm liking kvm as much.

 

perhaps if there was some good information about conversion, it would ease people's fears.

[morbidpete]

I'm not really reliant on Xen over KVM. I have no clue how to make a docker image. But that's irrelevant as my main reason for a VM is MediaCenterMaster and that's windows only. I do not pass any devices through. So KVM will be just fine for me. Using acronis to clone the machine and use universal restore to get It running in KVM. Plus VT-D and Xen has caused an issue in my latest build with my i5-4690K, so I lost support anyway even If I wanted to pass through a device.

[jonp]

i feel a schism coming, lol.

 

my first foray's into VM's were with xen and I liked it, i like xencenter in particular, but i'm liking kvm as much.

 

perhaps if there was some good information about conversion, it would ease people's fears.

 

The reality is that converting from Xen to KVM is dependent on whether we're talking about HVM guests (mainly Windows) or paravirtualized guests (Linux-only).  With PV guests, converting is more involved.  I added the steps required to the wiki:

 

http://lime-technology.com/wiki/index.php/UnRAID_Manual_6#Converting_VMs_from_Xen_to_KVM

 

It ultimately links to a guide on OpenSUSE's site, but as I indicate in the wiki, only certain steps need be performed.  For Windows-based Xen VMs, the process is going to be a little more involved, but I'm working on a guide.

[GHunter]

The only reason I can think of for wanting Xen over KVM is that it runs Windows 7 in a VM rock solid (this could be true for other OS's). I don't have any pass-through experience with it though as I never rolled back my Xen version. Even today I started up a new Win7 VM under beta 15 and I still get a random BSOD.

 

But I'm still sold on KVM. I'll buy more Windows 8.1 licenses if I have to.

 

Gary

[jonp]

The only reason I can think of for wanting Xen over KVM is that it runs Windows 7 in a VM rock solid (this could be true for other OS's). I don't have any pass-through experience with it though as I never rolled back my Xen version. Even today I started up a new Win7 VM under beta 15 and I still get a random BSOD.

 

But I'm still sold on KVM. I'll buy more Windows 8.1 licenses if I have to.

 

Gary

Well I think windows 7 works solid on KVM, but pass through may not. UEFI isn't technically supported in Windows 7 either, although some have found ways to modify their install media to support it.

[garycase]

I don't see any reason for UnRAID to support both hypervisors.    I'd put your effort into making the VM Manager for KVM as "clean" and robust as you can so it's easy to create and manage VM's.    It should be as easy to do this on UnRAID as it is in VMware Workstation.

 

There's a bit of a learning curve anytime you move between hypervisors ... but the overall concept isn't that much different and shouldn't be a big issue.    I've used Virtual PC, VirtualBox, VMware Workstation, ESXi, Xen, and a few others over the years ... and while the vast majority of my VM's are current in Workstation, I plan to migrate (or rebuild) a LOT of them into KVM once v6 is released and I build a new system for it.

 

[Interstellar]

Currently run a web server with a NIC passed through.

 

It works and has done for the past year.

 

If it ain't broke...

[garycase]

Currently run a web server with a NIC passed through.

 

It works and has done for the past year.

 

If it ain't broke...

 

... then don't fix it 

 

I agree.  On the other hand, that doesn't mean LimeTech needs to expend resources to support two different hypervisors.    Xen will still run ... there just won't be subsequent enhancements in UnRAID.

 

I don't have any issue with LimeTech support both if they choose to => my opinion is, however, that there's no reason to expend the resources to support both; and KVM seems like the better choice if they're only going to support one.

 

[meep]

I'm in a similar boat to Concorde Rules.

 

Even though it's Beta, I've been running v6 on my production server since around b4 or b5. A single unRAID box, aided the magic of virtualisation, now supports the following;

 

-whole house PVR & media

-whole house audio

-whole house storage

 

It's testament to the solidity of the unRIAD/Xen solution that I've come to rely on it so much. I run mostly Windows VMs as central PVR/media server (Win7) and HTPCs (Win8). (There's a Netrunner Manjaro workstation in there as well). I've passed through the following to the various VMs;

 

-3x AMD GPUs

-m-audio Delta 410 and 1010LT PCI multichannel audio cards

-Digital Devices Octopus tuner host card + 6x DVB-T / DVB-S2 tuners)

-Multiple USB Controllers

-Multiple discrete USB devices

 

(Such is my love of passthrough, I've added a PCI/PCIe expansion chassis to allow even more devices than my MB supports!!)

 

I know I've expressed concerns about migrating these VMs from Xen to KVM but, thinking about it, I'd probably just set them up with fresh installs. My bigger worry would be the compatibility and continued pass-through support for all of the above.

 

I'm concerned when I read notes around IOMMU Grouping in KVM and comments like this from jonp;

 

"I've had much easier success with NVIDIA devices that have UEFI BIOS than AMD.  I actually don't have any AMD cards that have UEFI capability.  Even a modern R9-290 I got from ASUS about a year ago doesn't have UEFI support.  With NVIDIA, most devices as of the 650 series and up seem to have it, and I've had particularly good luck with EVGA."

 

(https://lime-technology.com/forum/index.php?topic=39343.msg367341#msg367341)

 

I have an investment in AMD GPUs and don't really want to find myself in a position where I need to replace them with nVidia versions. So I'll need lots and lots of testing not only to get to grips with the KVM way, but to iron out any hardware issues.

 

Overall thought, I guess we're moving to a KVM world and the best way to address the above is to suck it and see. As it's a system that all the family increasingly relies on, finding the downtime to tinker and fettle is always difficult. In fairness, I have redundant systems for TV and Storage but it's still a royal PITA.

 

So all I'd ask would be that LimeTech retains the Xen boot option for the foreseeable future to give us time to test and transition.

 

In my use case, I have no real interest in Docker - I can run any app I wish in one of my Windows or Linux VMs.

 

Keep up the good work.

 

Peter

 

[garycase]

Sounds like a VERY nice setup.  I'm sure that any changes in the hypervisors will be AFTER v6 final is released; so as a minimum you should be able to keep everything as it is for the foreseeable future.

 

As a matter of interest, how do you connect the VM's to the display devices around the house?    Or is everything co-located?

 

[meep]

Sounds like a VERY nice setup.  I'm sure that any changes in the hypervisors will be AFTER v6 final is released; so as a minimum you should be able to keep everything as it is for the foreseeable future.

 

As a matter of interest, how do you connect the VM's to the display devices around the house?    Or is everything co-located?

 

Hi Gary

 

I'm fortunate in that I can locate my server in an attic space that's within easy reach of all the required TV points (either adjacent or directly downstairs)

 

I run max 20' HDMI and USB cables to each location. I've found TVs to be fine but some computer monitors can be sensitive to the long runs.

 

Peter

[dlandon]

I too have invested in Xen VMs that are now an integral part of my work and home life.  While I would probably move on to KVM once it is more mature in unRAID, I will need time to move or re-create my VMs in KVM.  It's not so much the VMs themselves, but the setup, configuration, and migration of the data that is of a concern to me in migrating them.  Once migrated, what issues will I have to sort out (there will be issues) and will they be as rock solid as they have in Xen.  I have passed through an Intel IGPU in Xen that I don't believe has done by anyone in KVM.  I'm also hearing about this particular hardware works best with KVM, etc.  I can't see me re-investing in a lot of new hardware when what I have is working just fine.

 

What is not being said here is the timing of Xen removal.  In a PM to me, Jonp is suggesting that Xen be removed by V6 final.  I suggested to him that they let it stand in its current form so we have time to migrate to KVM and I got push-back that it would be immediate.  Ouch!  Can we not have some time to make the change?

[meep]

Well, in terms of timing, if I'm going to recreate VMs in KVM, I'd probably make them Windows 10 (since Windows 7 seems to have UEFI compatibility issues).

 

Therefore, it would be great if Xen could hang in there until the end of July when Win10 is due for releaeor, more realistically, Autumn timeframe to allow for migration & testing.

 

There's nothing like a hard deadline to get things done!

 

 

[dlandon]

I won't have the time for this effort until the winter. If LT insists on removing Xen, I'll have to stay on the last version supporting Xen and forego any enhancements to V6 until I can invest the time to make the move.

[garycase]

I have no insight into what LT might be planning, but I seriously doubt Xen will be removed before v6 final; and it'll almost certainly be well into the fall before there's a v6.1, so I think there's little concern about lack of support this year ... you guys should have plenty of time to move to KVM if that's the course LT takes (which seems likely).

 

Peter => You're fortunate indeed that you don't need any runs longer than 20' (that's actually not long at all).  To briefly hijack this thread, I'm going to describe what I'm considering and see if you have any comments based on your experience ...

 

I have 3 independent HTPC's => the primary one has 8 tuners (4 dual-tuner Hauppauge 2250's),  8TB of storage for recordings, and is always on (it does all the recording).    The other 2 are only on when we're watching their attached TVs ... which really means we're watching either something that was recorded on the main HTPC or a movie from my media server (an UnRAID server that's also always on).    I've got Cat-6 runs throughout the house, so everything's connected; but if I ran HDMI to all the TV's the runs would be over 100', so I'll just stay with the independent systems.

 

The primary HTPC only has 8TB of storage; but I move anything we want to keep to an UnRAID server that has FAR more (only 24TB at the moment, but easily expandable if needed).    I've been toying with the idea of making that box a v6 UnRAID box and installing Windows 7 (or 10) in a VM to manage the recordings;  but I'm not sure if the Hauppauge cards will pass through okay.    I'll probably test that this fall, as I can't really do much testing on the "live" system, since my wife seems to ALWAYS have something scheduled to record    [You'd be surprised how much you can record when you can record 8 channels at once and have virtually unlimited storage => we don't come CLOSE to watching it all ... but if somebody says "Have you seen this new series?" we can almost always say, "Oh, we haven't, but we have it all recorded, so we'll have a look !!"]

 

Did you have any issues passing tuners through?  ... and were there any bandwidth constraints if/when you did that?  How many channels do you record at once?    I presume you record to a dedicated non-array drive ... is that correct?    I'm thinking that if I make the main HTPC an UnRAID box, I'll have everything record to a 1TB SSD, and then move all of a day's recordings to the array overnight.  That would certainly eliminate bandwidth issues for recordings; and 1TB is FAR more than we'd ever record in a day => I may even make the moves only apply to recordings more than a few days old, so stuff we record/watch/delete (news programs, talk shows, etc.) wouldn't get moved if we'd already watched them.

 

 

 

[jonp]

First and foremost, I want to thank everyone for their feedback and to keep it coming.  I realize that to eliminate support for Xen would be an inconvenience for some, but that's not actually the point of this thread.

 

The point of the thread is to hear from folks as to what unique benefit Xen can provide that warrants it's continued support.

 

As far as dlandon's comment about my PM to him, he is correct in that it is the plan to remove xen support before final.

 

The sheer cost in time/resources to support Xen isn't something we can justify. For us to keep it in for 6.0 final would mean a massive increase in cost to testing, documentation, development, and support.

 

As dlandon said, for users that do not want to lose their current setup, remaining on a beta version for a prolonged period may be a viable path.

 

Also, with respect to AMD GPUs, I have gotten multiple AMD GPUs to work with KVM, but I just don't have any to test with that have UEFI support. That's not the end of the world. All Xen setups with GPU pass through don't leverage UEFI either.

 

Lastly, testing this isn't very difficult.  Reboot the server into non-Xen mode on beta 15 and test out KVM with your hardware to see how it works for you.

 

I realize the inconvenience this will cause some that rely upon Xen, but again, it is just not in the cards for us to continue supporting Xen unless someone here can show us a viable unique use case.

[garycase]

... it is the plan to remove xen support before final.

 

Ouch !!  I'm not a Xen user, but I'm nevertheless surprised you're doing that this close to final.  I'd have thought that would have been perhaps an announced v6.1 change => but that Xen would still be available for those who have invested significant time in Xen setups (such as dlandon & meep above).

 

I can only surmise from this that perhaps some of the remaining issues for the RC are Xen issues.  If not, I don't understand why you'd do this now.

 

 

 

[jonp]

... it is the plan to remove xen support before final.

 

Ouch !!  I'm not a Xen user, but I'm nevertheless surprised you're doing that this close to final.  I'd have thought that would have been perhaps an announced v6.1 change => but that Xen would still be available for those who have invested significant time in Xen setups (such as dlandon & meep above).

 

I can only surmise from this that perhaps some of the remaining issues for the RC are Xen issues.  If not, I don't understand why you'd do this now.

 

That is correct. We were hopeful of merging support for xen into our libvirt vm manager, but in recent testing, this will not work. It misses some key features like usb assignment that make it a non-starter.

 

[mr-hexen]

Then why not stop development of xen but leave it as is in the current b15?

 

LT incurs no further development costs but users can continue using it for the foreseeable future.

[sparklyballs]

If you can get a ubuntu desktop up and running with OVMF that would be nice, lol.

 

 

 

[mace]

I've been running a Xen VM from v6b4 (and am still on b4 as well).

There's nothing that Xen gives me that another hypervisor wouldn't, so I would probably just do the VM again from scratch rather than convert it.

 

[sparklyballs]

If you can get a ubuntu desktop up and running with OVMF that would be nice, lol.

 

 

should note, i'm purely being selfish here, hee hee.

 

it's not something which will hold anything up because seabios definitely works with ubuntu.

[Interstellar]

Bottom line is:

 

If you can swap us over to KVM with ideally none, or minimal time required for tweaking and no loss of performance then fine.

 

I don't want to be told that my current VM is not going to be KVM compatible, that wouldn't be nice given from what I recall Xen was picked first when this all got started!!

[jonp]

Bottom line is:

 

If you can swap us over to KVM with ideally none, or minimal time required for tweaking and no loss of performance then fine.

 

I don't want to be told that my current VM is not going to be KVM compatible, that wouldn't be nice given from what I recall Xen was picked first when this all got started!!

 

The amount of effort required will depend on the type of VMs you have.  If you have paravirtualized guests, converting them to HVM is a little more involved, but it is possible.  If you were using Windows-based guests or passing through PCI devices, then you have an HVM setup, and that is going to be a lot easier to convert.

 

As for performance, if anything, you should see an improvement converting HVM workloads from Xen to KVM.  KVM's virtio drivers seem to be much better than Xen's GPLPV drivers and are continuing to get refined and updated (most recently they were updated in January of this year).

 

As far as your VM setup not being KVM compatible, if you have Linux-based VMs today that are not doing GPU pass through, Docker is the best solution for those.  If you have GPU pass through to any VMs, KVM should be able to handle it, but there are a few cases where it may not.  All you can do is try and see.

 

See the blog post referenced in my forum signature for the complete story on our journey with virtualization.  I think it explains why we had to pivot and how Docker has eliminated the purpose we were hoping Xen would serve.