64 Bit unRAID running natively on Arch Linux with full hypervisor support

[Ford Prefect]

Thanks for sharing your thoughts, TOM

 

let me comment and share some first thoughts on this one.

Is this possible to do?  Add a crypto layer between md-devivces and mounted disks, and it's the mounted disks (or shares) that get exported to client VM's.  This way all client VM disk contents become encrypted "automatically".

 

Yes, you could use LUKS/dm-crypt to form that layer.

The devices will need to get initialized/encrypted with a key and a device name for the device when opened/unlocked, like "cryptsetup /dev/md0 unraid-enc-open-md0"

After the devices are encrypted they need to be opened with the key in order to be accessed.

After being opened with the key, the unencrypted devices are available under their other name like "/dev/mapper/unraid-enc-open-md0"

This mapped device then can be used to create a FS.

...this is all about semantics in device-names and how md-module, LUKS/dm-crypt and emhttp handles them.

 

But obviously you still need to maintain the mapping between raw real disk , md-device and encrypted device as you want to gather info from all these, like SMART-state, FS state, ....

 

And what are your thoughts on the term  "automatically", that you used.?

IMHO it is not a god idea to define a key silently...if something goes wrong and the key is lost, the data is no longer retrievable.

And you'll also need some user comfort when encrypting many devices...no use in entering the key 24 times upon boot for a maxed array..

There are some different ways to do that...I think we will come up with something, depending on how integrating the workflow in emhttp will look like.

 

[limetech]

A lot of people also believe in the following rule:

 

unRAID running in VM with PCI Passthrough on ESXi / Xen / KVM / etc. is SAFE but Xen / KVM running on unRAID is NOT SAFE.

 

I don't believe in such a rule.  I'm thinking Xen/KVM running on unRaid is the way to go.  I think we are just trying to hash out what constitutes "unRaid" in such a configuration.  I'm thinking it can be much the same as it is now, with additional modules needed to support Xen/KVM.  Call that the "storage/VM" layer.  Now if you want other linux features, such as a window manager, xbmc instances, whatever, then let them run in VM's.  Am I missing something?

 

Not sure how such a rule every started. I haven't been in a Fortune 500 Enterprise Environment in 30+ years where they would dream of doing it that way. In fact, they spend / invest a lot of money to do the exact opposite.

 

Guess users with little to no Linux / IT / Storage Experience with their home NAS devices full of Movies, TV Shows, etc. have it all figured out. Clearly, Fortune 500 companies who spend millions on IT professionals, services, consultants, have 10+ Million dollar ERP Applications and Billions in Revenue to lose need to get with the times and put all their Storage in VMs with PCI Passthrough. Otherwise they will lose their data.

That's a bit of a grumpy thing to say (but no skin off my back).  People are learning and coming up to speed and we value your input as someone who clearly knows what they're talking about.  Remember, forum threads are notorious for miscommunication.

 

Last project I worked on, we consolidated 1,200+ Mission Critical Servers (most of with had their own storage) that were spread-out throughout the Enterprise and located at various branches and Data Centers. Maintaining compliance with Sarbanes Oxley and all the other Financial Regulations we had to be in compliance with...  We ended up using ESXi and EMC as a solution. Not a single one of the VMs was it even discussed or thought of to use PCI pass-through for storage. Perhaps it had something to do with the data that was mission critical and worth millions of dollars the customer and the consultants didn't want to put it at risk of being lost, corrupted, etc. when using the PCI Passthrough method.

 

To listen to some of the people here... Maybe the company and us consultants had it all wrong.

Sometimes people just react to an attitude of "Hey I know what's best & it's my way or the highway!".  Again, a risk of forum thread communication.

 

By having your Data / Drives sit outside the VM... Technically it is considered more secure, safe, reliable for the integrity of the data, support, uptime, etc. If you think otherwise, I can post some whitepapers on the subject and we can discuss those. I think it's common sense but maybe I am in the minority.

 

Point is, If we are going to use safety and integrity of the data as the most important part of the equation... Let's at least be honest. Running unRAID in a VM with PCI Passthrough is considered less safe than doing it the other way around.

 

You make a good point, and I can see how that would be better.  Actually I never really liked "pass-through" of storage to unRaid in a VM - always struck me as "fragile".

 

 

[WeeboTech]

By having your Data / Drives sit outside the VM... Technically it is considered more secure, safe, reliable for the integrity of the data, support, uptime, etc. If you think otherwise, I can post some whitepapers on the subject and we can discuss those. I think it's common sense but maybe I am in the minority.

 

Point is, If we are going to use safety and integrity of the data as the most important part of the equation... Let's at least be honest. Running unRAID in a VM with PCI Passthrough is considered less safe than doing it the other way around.

 

You make a good point, and I can see how that would be better.  Actually I never really liked "pass-through" of storage to unRaid in a VM - always struck me as "fragile".

 

 

I though what attracted people to PCI pass through was accessing the drives with complete control.

spin and smart control without interfering with other hardware.

No hard locked drive configuration at the hypervisor level. For most of the hobbyists, that works well for them.

[Ford Prefect]

I though what attracted people to PCI pass through was accessing the drives with complete control.

spin and smart control without interfering with other hardware.

No hard locked drive configuration at the hypervisor level. For most of the hobbyists, that works well for them.

 

IMHO this is all about priority of requirements and possible features to implement them.

This is only applicable IF you have put the capability for virtualization at the top rank of what you want.

Virtualization first, unRAD/NAS second, all on a single host comes third.

 

With exiting bare metal unRAID, options for this are limited.

Virtualizing unRAID is possible with a good hypervisor and the right hardware.

So this passthrough feature attracted people because the "know" that unRAID in a VM is a risk...and even more so with an additional

layer between the unRAID md-module and the disks (when using RDM)....passthrough just mitigates this.

[grumpybutfun]

I though what attracted people to PCI pass through was accessing the drives with complete control.

spin and smart control without interfering with other hardware.

No hard locked drive configuration at the hypervisor level. For most of the hobbyists, that works well for them.

 

I don't get a vote and as I stated earlier, if that is what the majority wants (which it sounds like it is)....

 

If I was Tom, I wouldn't jump through all the hoops to make it a reality.

 

If unRAID continues with Slackware and the bzoot method, It's going to take one heck of a plugin writer to write one hell of a plugin to do everything that is needed to make KVM / Xen work, be user friendly and "useable" for a novice.

 

If I was Tom, I am not sure I would go down that road since there is only a handful of users who seem to want this functionality.

[Ford Prefect]

Ok so what I'm hearing is, for a CentOS-VM with Plex installed in it (because that is one of the Linux installs Plex offers in addition to Ubuntu and Fedora), I should not really have to worry about pass-through since Plex just needs a place in the VM-OS to store its database and network access.  And the same can be said for just about all the other typical plug-ins we run, or want to run, right now like SANZB, Couchpotato, uTorrent, MakeMKV (ok this might need direct access to the BD-drive?), Handbrake ... 

 

But things like using it as an HTPC with XBMC connected to a TV will require video card pass-through.

 

And of course this one reason why we'd rather have UnRaid as part of the host vice running as a VM since it opens up the hardware we can reliably use.  Right?

 

...yup...that's the idea.

[apgood]

Sounds like Tom and grumpy both agree or at least lean towards unraid being the hypervisor, but there is still discussion on how this unRAID hypervisor would / could be implemented.

 

Can see the attraction of both points of view:

 

1. Unraid hypervisor as appliance: Similar to current unraid implantation model. Benefits of things like support (e.g. safemode boot with just core product inc hypervisor but no plugins / addons), needs less Linux know how than option two (e.g. no real extra knowledge need to current version if just used for basic media storage).

 

Cons : not as flexible can't take advantage of things like onboard GPU?

 

2. unraid as a multipurpose server. All the many benefits badger and grumpy have mentioned.

 

Cons:  support and troubleshooting harder for Tom as possible fragmentation of core modules and even kernels across user base.  No longer an appliance experience for people that just want a PC based nas appliance.

 

Naturally this isn't an exhaustive list of pros and cons just a quick summary.

 

[jbartlett]

If I was you, would I jump through all the hoops for 51 people who responded to the vote in this thread?

 

Keep in mind that there have been people who expressed that they're interested in this but haven't voted because they either don't have enough information to vote one way or the other or simply don't care which OS is used - myself being one.

 

This thread is only 2 days old.

 

I'm willing to bet that there's at least 100x the number of  forum readers who would be interested in this if they knew about it - but still may not vote.

[Ford Prefect]

...when unRAID will open up for native Hypervisor and virtualization support, all the VM data and configs need to go somewhere as well.

 

I don't see this to be integrated as a plugin....there are other and better solutions out there.

 

I can understand the requirement to keep the distro clean, in a snapshot with well known content.

Maybe one needs to rethink the hybrid approach.

 

Why not have a "pro", appliance version? ... and an enthusiast/community version?

..with a different lifecycle and licensing and support (...and cost) scheme?

I can see the enthusiast version to be released more often...like being a test-repo for the pro stream...only stable stuff will be moved into pro/appliance stream by TOM.

 

Will need some help from TOM to kick this of..especially with emhttp we need some more generic integration features (not to say a public API), but AFAIU

there are some thought from TOM about this already (senn in the encryption layer discussion thread).

[grumpybutfun]

Keep in mind that there have been people who expressed that they're interested in this but haven't voted because they either don't have enough information to vote one way or the other or simply don't care which OS is used - myself being one.

 

This thread is only 2 days old.

 

I'm willing to bet that there's at least 100x the number of  forum readers who would be interested in this if they knew about it - but still may not vote.

 

How about this as a compromise...

 

Being the poor college student that Ironic is, he puts together an Slackware 14.1 64 bit with unRAID Xen and KVM ISO (since the unRAID wiki is complicated and incomplete).

 

During the install (to a flash drive or Hard Drive) you select if you want a KDE Linux Desktop (looks and works much like Windows), XBMC or just a terminal screen.

 

He will also provide a repo where you can do...

 

slackpkg install mysql couchpotato plexmediasever etc.

 

If you want to install various packages. Since the Slackware repo sucks he can add the popular ones or users requests so none of you have to compile stuff.

 

Would any of you guys be willing to donate money to Ironic if he does that and maintains a repo with software which it updates with new versions as they come out?

[sparklyballs]

I'm interested in this, but i'm not voting because my knowledge of linux and the pros and cons of the various distro's is limited.

I'm really good at copy and paste though !!!!!

[dirtysanchez]

If I was you, would I jump through all the hoops for 51 people who responded to the vote in this thread?

 

Keep in mind that there have been people who expressed that they're interested in this but haven't voted because they either don't have enough information to vote one way or the other or simply don't care which OS is used - myself being one.

 

This thread is only 2 days old.

 

I'm willing to bet that there's at least 100x the number of  forum readers who would be interested in this if they knew about it - but still may not vote.

 

Totally agree. Count me as one that's possibly interested in this project depending on where it goes, but isn't Linux savvy enough to have a say as to what distro should be used.

 

It's also obvious that some here want changes immediately, while others want to carefully consider all sides and go from there. IMHO nothing good can be gained from rushing this, but then I'm no Linux guru either.  I understand that for the most part this is just "packaging" unRaid differently, with at most some minor changes, but I still think it deserves to be carefully thought through from all sides.

[Ford Prefect]

How about this as a compromise...

 

Being the poor college student that Ironic is, he puts together an Slackware 14.1 64 bit with unRAID Xen and KVM ISO (since the unRAID wiki is complicated and incomplete).

 

During the install (to a flash drive or Hard Drive) you select if you want a Linux Desktop, XBMC or just a terminal screen.

 

He will also provide a repo where you can do...

 

slackpkg install mysql couchpotato plexmediasever etc.

 

If you want to install various packages. Since the Slackware repo sucks he can add the popular ones or users requests so none of you have to compile stuff.

 

Would any of you guys be willing to donate money to Ironic if he does that and maintains a repo with software which it updates with new versions as they come out?

 

...nice idea...but...based on ...Slack... ?!?..I think you lost me there.

If you want to put it to a test, what kind of setup would be most deserved?

- unRAID, native with XEN/KVM modules-> check

- VM management via WebUI - or via VirtManager/local GUI -> check as this is minimal

- with native XBMC - most likely

- with ......... -> what distro is most complete and stable?

 

I'd take that and promise just that if you want to take it for a test and ironic to get some funds/compensation for his time.

I am willing to donate some bucks just for the sake of it not being based on Slack 

[WeeboTech]

 

I would be willing to contribute monetarily and/or technical assistance.

My request would be that the XEN/KVM kernel specific settings be brought to the table for Tom to incorporate.

[grumpybutfun]

...nice idea...but...based on ...Slack... ?!?..I think you lost me there.

If you want to put it to a test, what kind of setup would be most deserved?

- unRAID, native with XEN/KVM modules-> check

- VM management via WebUI - or via VirtManager/local GUI -> check as this is minimal

- with native XBMC - most likely

- with ......... -> what distro is most complete and stable?

 

I'd take that and promise just that if you want to take it for a test and ironic to get some funds/compensation for his time.

I am willing to donate some bucks just for the sake of it not being based on Slack 

 

Tom didn't seem to fond of an unRAID Distro.

 

Since unRAID has instructions on the wiki for installing unRAID on a Full Version of Slackware... I thought it would keep Tom happy and give users who want more flexibility an option.

 

They could configure the VMs and manage them via a Linux Desktop and choose whether or not it boots straight into the Desktop or XBMC. If you need to stop or start a VM, switch over to the desktop, etc. You run X Applications (virt-manager) through SSH with X Forwarding. Plus you can always VNC to the Desktop if the computer sits in a closet. Not perfect but it works well until a WebGUI for KVM / Xen is nailed down.

 

Just thinking out loud and trying to keep Tom happy as well.

[limetech]

If you have Xen or KVM running with a way to startup VM's, what's the point in having XBMC "native"?  Why not just run it in a VM, or how about running XBMCbuntu in a VM?

 

Why do you need a desktop?  Again, just run <your linux distro of choice> in a VM.  Isn't the whole point of this little discussion to do exactly this?

 

As for managing the host itself - can't this be done using the shell to start?  Are there any web-based management tools for Xen and/or KVM.

 

Which is better to use, Xen or KVM?  Is it necessary to have both?

[limetech]

I don't know about you guys... but I need a drink.  Back later.

[grumpybutfun]

If you have Xen or KVM running with a way to startup VM's, what's the point in having XBMC "native"?  Why not just run it in a VM, or how about running XBMCbuntu in a VM?

 

My Server sits near a TV.  I have 2 video cards and the onboard GPU all of which drive 2 XBMCs (throughout my house) and a Windows VM (in my office).

 

Why waste a perfectly good GPU / Video Card with a blinking cursor that says login (which hardly anyone does)?

 

Also, there are plenty of users who do not have a CPU, Motherboard or Video Card that can do passthrough. If I ran a full version of Slackware, used the on board video for XBMC... I still can run VMs for various apps like Sickbeard, Owncloud, phpmyadmin, Wordpress, etc.

 

I am going to go out on a limb and guess that 70% or more of the customers who have purchased unRAID in the last 3 or 4 years learned about unRAID from the XBMC, Plex, XBMCbuntu, Media Portal, OpenELEC, etc. forums. Therefore, getting the most out of their GPU / Video Cards is important to them.

 

Why do you need a desktop?  Again, just run <your linux distro of choice> in a VM.  Isn't the whole point of this little discussion to do exactly this?

 

As for managing the VM itself - can't this be done using the shell to start?  Are there any web-based management tools for Xen and/or KVM.

 

That is a long story. Simple answer is, no.

 

Which is better to use, Xen or KVM?  Is it necessary to have both?

 

If I was you, why manage three things (unRAID Kernel, Xen kernel and Xen package) when you can use KVM built into the Linux Kernel? It does everything Xen can do (in fact it's slightly ahead) and based on my experience in very large scale deployments... The "speed" is basically the same. In my limited testing using unRAID in a VM on Xen and KVM... there was no difference speed wise.

[limetech]

Why waste a perfectly good GPU / Video Card with a blinking cursor that says login (which hardly anyone does)?

Can't the on-board Video get allocated to a VM via passthrough?

 

Also, there are plenty of users who do not have a CPU, Motherboard or Video Card that can do passthrough. If I ran a full version of Slackware, used the on board video for XBMC... I still can run VMs for various apps like Sickbeard, Owncloud, phpmyadmin, Wordpress, etc.

 

I am going to go out on a limb and guess that 70% or more of the customers who have purchased unRAID in the last 3 or 4 years learned about unRAID from the XBMC, Plex, XBMCbuntu, Media Portal, OpenELEC, etc. forums. Therefore, getting the most out of their GPU / Video Cards is important to them.

Sure but I bet if the functionality was there lots of those folks would upgrade.

 

That is a long story. Simple answer is, no.

You're saying it's not possible to fire up VM's via the command line?  How hard can it be?

 

If I was you, why manage three things (unRAID Kernel, Xen kernel and Xen package) when you can use KVM built into the Linux Kernel? It does everything Xen can do (in fact it's slightly ahead) and based on my experience in very large scale deployments... The "speed" is basically the same. In my limited testing using unRAID in a VM on Xen and KVM... there was no difference speed wise.

Your recommendation is to implement KVM and forget about Xen at least for now?

[grumpybutfun]

Can't the on-board Video get allocated to a VM via passthrough?

 

It's complicated.

 

Sure but I bet if the functionality was there lots of those folks would upgrade.

 

I agree but what is the harm in doing the following...

 

1. Enabling the Radeon, Nouveau and Intel Drivers in the kernel.

 

Within the bzroot have the packages in a folder somewhere (like you did for mysql)

 

2. XBMC (which updates once a year... Twice at most that I have seen since version 9)

 

3. Xorg (which also doesn't have that many updates and most users are running Xorg-Server 1.13 anyway which is a few versions back)

 

This take up MBs of space and if the user wanted to run it, they do it with the go file. Not to mention it would be a very easy plugin to write which I am sure someone would do.

 

The plugin would handle xorg.conf and XBMC to point somewhere on the cache drive for the configs / settings / etc.

 

We are talking about 50 MBs or so of space without it being compressed.

 

How many people are using 256mb USB Flash Drives out there?

 

I will bet you A LOT of money that doing the above alone will have users praising your name, singing songs about you and some will even name their son or daughter Tom in honor of you.

 

You're saying it's not possible to fire up VM's via the command line?  How hard can it be?

 

I misunderstood. Yes you can.

 

WebGui is what I thought you asked about.

 

Your recommendation is to implement KVM and forget about Xen at least for now?

 

I doubt you would get a single complaint. A Hypervisor is a hypervisor. It just so happens that KVM and Xen can be used in the case. Xen for you, would require a Xen Kernel and Xen Package. With KVM it's backed into Linux Kernel and the QEMU package (which the plugin would download and install... it updates too often).

[nicinabox]

 

He will also provide a repo where you can do...

 

slackpkg install mysql couchpotato plexmediasever etc.

 

If you want to install various packages. Since the Slackware repo sucks he can add the popular ones or users requests so none of you have to compile stuff.

 

Would any of you guys be willing to donate money to Ironic if he does that and maintains a repo with software which it updates with new versions as they come out?

 

While this is somewhat in the right direction, this type of gatekeepery is not conducive to distributed growth. It keeps software in silos, unable to be reviewed, changed, or improved by anyone but the creator.

 

What's really needed is a package distribution system that doesn't rely on one person to function. Developers should be able to publish and distribute packages easily, and users consume them in the manner above.

 

Full disclosure: I've been working on such a project recently (boiler) which is modeled after Bower, but is not yet ready for production.

[limetech]

Can't the on-board Video get allocated to a VM via passthrough?

 

It's complicated.

Right, but that's how badger earns the big bucks 

 

I agree but what is the harm in doing the following...

 

...

 

How many people are using 256mb USB Flash Drives out there?

Sure all those things can be added easily.  It's not the flash size I'm worried about, it's the size of 'bzroot' I try to keep down.

 

I will bet you A LOT of money that doing the above alone will have users praising your name, singing songs about you and some will even name their son or daughter Tom in honor of you.

(but all that can be done with a plugin)

 

Your recommendation is to implement KVM and forget about Xen at least for now?

 

I doubt you would get a single complaint. A Hypervisor is a hypervisor. It just so happens that KVM and Xen can be used in the case. Xen for you, would require a Xen Kernel and Xen Package. With KVM it's backed into Linux Kernel and the QEMU package (which the plugin would download and install... it updates too often).

[jumperalex]

I though what attracted people to PCI pass through was accessing the drives with complete control.

spin and smart control without interfering with other hardware.

No hard locked drive configuration at the hypervisor level. For most of the hobbyists, that works well for them.

 

[scratching head] ok I'm probably a bit confused (no duh).  But I thought the order of "goodness" was:

 

1) Unraid as host with direct access to SATA, all other VM's go under that.

2) Unraid as VM with pci pass through access to SATA

3) Unraid as VM with some other virtualized access to SATA that either doesn't actually work, is slow, unreliable, or all three

 

The only reason pci pass through was ever considered "desirable" was because it was better than 3) and 1) was not even an option.

 

I ask because trying to follow everything that followed I feel like either the above was lost in the noise, or I've been woefully misunderstanding what Ironic has proposed, and grumpy has been trying to get us to understand, from the very start.

[grumpybutfun]

While this is somewhat in the right direction, this type of gatekeepery is not conducive to distributed growth. It keeps software in silos, unable to be reviewed, changed, or improved by anyone but the creator.

 

With all due respect, this is exactly how Linux Package Mangers handle it today.

 

Using Samba and Debian as an example:

 

http://packages.debian.org/wheezy/samba

 

You will notice there are in this case a team of people (Maintainers) who are solely responsible for their package and integration / consistency / updates / bug reports / reporting issues to the Linux Kernel and Samba folks / testing of Samba in Debian and all the various versions of Debian too. These people are experts on Samba, what's new, what bug fixes are needed, what patches may or may not be out there for security flaws, preparing it for the next version for release, preparing it for the next release of Debian, etc.

 

These people know more about Samba than you and I could ever hope too and how to make it work best... in this case within Debian.

 

What's really needed is a package distribution system that doesn't rely on one person to function. Developers should be able to publish and distribute packages easily, and users consume them in the manner above.

 

Full disclosure: I've been working on such a project recently (boiler) which is modeled after Bower, but is not yet ready for production.

 

If we are talking about plugins, custom apps or custom compiles of apps like XBMC, Wordpress, etc. then I agree with you.

 

But if you are talking about CORE Linux Packages and running Server Grade NFS, Samba, Python, Curl, etc. you are not going to get a more stable / reliable Samba, NFS, Python, etc. package than the one in the [insert Linux Distro] package manager.

[limetech]

I though what attracted people to PCI pass through was accessing the drives with complete control.

spin and smart control without interfering with other hardware.

No hard locked drive configuration at the hypervisor level. For most of the hobbyists, that works well for them.

 

[scratching head] ok I'm probably a bit confused (no duh).  But I thought the order of "goodness" was:

 

1) Unraid as host with direct access to SATA, all other VM's go under that.

2) Unraid as VM with pci pass through access to SATA

3) Unraid as VM with some other virtualized access to SATA that either doesn't actually work, is slow, unreliable, or all three

 

The only reason pci pass through was ever considered "desirable" was because it was better than 3) and 1) was not even an option.

 

I ask because trying to follow everything that followed I feel like either the above was lost in the noise, or I've been woefully misunderstanding what Ironic has proposed, and grumpy has been trying to get us to understand, from the very start.

I'm leaning toward 1) now.  Makes the most sense.