Xen/unRAID-6 Discussion

[dalben]

How about if there is a plugin manager and one of your options is "Install XBMCbuntu in a VM"?

 

Yep.  That would work.

 

There are many ways of skinning this cat I guess.  Installing a pre-configured VM that you know will work with unRAID without too much fiddling would be as useful as a plugin.  Ideally the config settings for the XBMC and other apps installed via VM can still be done via the same console/webgui like plugins are now.

[SchoolBusDriver]

Yes, you can have VM Appliances. It's is very easy to do.

 

Guys, we aren't blazing new trails or reinventing the wheel. What we are doing is applying a "Best of Breed" Practice that Enterprises, Large, Medium and Small Business do with our Home Server(s).

 

I thought we have already discussed this earlier in the original beta release thread and the one in the Virtualization forum.

 

An Example of what I am talking about:

 

Turnkey Linux Virtual Appliance Library

 

VMWare Virtual Appliance MarketPlace

 

Slacket Virtual Images and Templates

 

Option A

 

I could create a VM Appliance with CouchPotato, Sickbeard, Sabnzbd, etc. in 30 minutes or less and post a link here. All you would have to do... Use the cfg file and the vhd / img / cow file I provide and away you go. Within the thread I tell you the ports of the various Apps / WebGUIs and you then go configure each App to work how you want on your system.

 

Option B

 

I break up with my GF, spend WEEKS / MONTHS writing all the various plugins where I get paid squat. The plugins are still going to be flaky and everytime unRAID or the various programs changed... I'd have to figure out what I needed to update / change and mess with the plugin more. Not to mention, I'd then have to write a book on how to install it and hope my plugins or someone elses does not break everything.

 

I'm not sure about the rest of you... but I am going with Option A. I choose it everyday of the week and twice on Sunday.

 

There are a lot of cool Apps that will blow your mind which I will create several VM Appliances for once we get the issues ironed out. I can pour my knowledge and experience into several VM appliances and do all the heavy lifting for you and provide a brief overview of how to access the configuration of the various Apps for you to customize to your liking. You update the Apps via a Package Manager in whatever Linux Distro I put them in and being the nice guy I am... I will write a damn good guide like my others explaining in GREAT detail how exactly to do that too. EASY for ME and EASY for YOU!

 

In case you are wondering... NO! I am not dumping my GF, and spending WEEKS / MONTHS writing plugins for the various Apps that you will find in my VM Appliances. No... not even if you pay me.

 

Virtualization isn't even 21st Century Technology... So either get with the times or stick with the limitations you have with unRAID 5.0 and plugins.

[SchoolBusDriver]

I think we all love the fact the array disks get spun down when not being used. Therefore the next most logical place for them is the cache drive, probably a SSD. Since the cache drive is not protected it will be nice to have a method to regularly back up (or snapshot) them to protected storage.

 

Otherwise, it reinforces the need for a mirrored cache system.

 

Your VMs can sleep just like your "bare metal" PCs can.

 

So you could use unRAID Drives and have everything sleep or stick with the Cache Drive... It's up to you.

 

Use WOL or the 5 or 6 other ways you can have your Linux OS "wake up" from sleep.

[jevans04]

I can't wait to ride the "short" bus to Option A.

I am standing at the bus stop (im)patiently waiting......

[SchoolBusDriver]

I can't wait to ride the "short" bus to Option A.

I am standing at the bus stop (im)patiently waiting......

 

[limetech]

I can't wait to ride the "short" bus to Option A.

I am standing at the bus stop (im)patiently waiting......

Working my a$$ off to get this done, trying not to get run over by any buses 

[madburg]

See the position being missed here is that some don't need and don't want to use anyone elses flavor of VM, and be depended on it, you get hit by a school bus tomorrow and they don't know what to do.

 

If someone wants to virtualize they have turnkey hypervisors for that today, right now. As one example, I have ESXi, I run various VMs, one of them is unRAID, I have a dedicated controller for it, and a dedicated controller for my VM's (VM Datastore) i picked the raid level I wanted, I back them up the way i want It doesn't affect unRAID nor unRAID performance. I spin up any OS I want, load whatever apps I want on VMs today, right now.

 

But to have to wait for someone to create a plugin for unRAID and it may or not work well and as easy as other turn key hypervisor already out there is questionable. I agree with dalben comment and believe in the unRAID I purchase for what it does being a good NAS which spins down drives.

 

For those who don't have the money/interest/knowledge to do something like the above to offload apps like sab, sickbeard, via VMs etc.. they want it to run on unRAID as a plugin, and with that, the hope was with a x64 bit version from Tom, it would be what is require to run those type of plugin and not run into memory issues as they currently are it the 32 bit version.

 

What it sound and looks like at this moment is, yeah here is the 64 bit version, but don't run plugins, run them with a VM through unRAID with no solid easy way to do so (as it stands). So I ask why? I have my chosen hypervisor today which I can spin up an VM OS i want with a few clicks and just want a 64 bit unRAID, to be able to say run cache-dirs on ALL the content and preclear drives at the same time, run parity check without things boming out.

 

I understand for the tinker and advance linux user this is great fun but it not for the general community. TO the point above if I only have 4-8 gb of ram I just want a few thing say like sab/sickbeard/couch to run on unRAID. Why cant have person run it as a plugin on a 64 bit version, why would they have to run a VM and allocate it memory to run those three apps, that just doesn't make sense.

 

P.S. I am not saying to ripe XEN out, if its a small footprint and makes advanced linux users happy to pay with, sure they can knock themselves out and play.

[jbartlett]

My inner geek wants to thank you all!

 

*rubs hands together* muhahahaha!

[jumperalex]

And we appreciate it Tom.  Thanks

 

And of course grumpy, ironic, and others, thanks for helping Tom and eventually us as well.

[BLKMGK]

A question: above it was mentioned hardware being passed through. In ESXi doing this means no other VM can have access, does Xen have this same limitation? I assume so, does that mean that the paravirtualized drivers are a near must? Does Windows have these as well under Xen or KVM? If I do passthrough of hardware is there an advantage to running a more capable video card? I've seen talk of virtualizing XBMC for instance, what's the front end for that at the TV? Guess that was more than one question ;-)

 

From the config posted above spinning a text file to launch a VM doesn't look too bad, a simple web GUI could produce that and even query the drive subsystem for parameters. What about performance monitoring, thoughts?

 

I too look forward to a cache setup that's running something like BTRFS and perhaps a diagonal parity drive for the rest of the array if it can be done without too great a performance hit! I'm excited!

 

Today I believe the cache drive can already be mirrored with a RAID card, yes? I don't currently run a cache, sounds like I need to soon! If a switch from ESXi I'll have a few SSD freed up ;-)

 

P.S. as to the point of some folks wanting to stick with ESXi or for those wanting to just run plugins - go for it. None of this precludes you from doing ANY of that! Now, you may find that the folks who have been building the Couch/Sick plugins no longer have an interest because they have a better way. If so you may have to begin building your own plugins or move up. I doubt that will happen and certainly not soon but if something WAY better comes along people will move to it. UnRAID never promised to be able to be much more than a secure place to store files, plugins have been icing on the top built by volunteers so IMO you can't cry too much IF those folks lose interest in the future....

 

 

Sent from my iPad using Tapatalk

[jbartlett]

See the position being missed here is that some don't need and don't want to use anyone elses flavor of VM, and be depended on it, you get hit by a school bus tomorrow and they don't know what to do.

 

My understanding is that you can boot the 64bit Unraid without Xen and install 64bit plugins the same way as you did with 32bit plugins on the 32bit version of unraid.

[madburg]

That was my understanding from Tom's first post clarifying things with this 6.0 beta. I just want to express that I hope that options remains through 6.0-Final and does not get removed. That flexibility would be greatly appreciated.

 

I appreciate and thankful that ssh, mailx and other improvements came with the 64 bit version, so I no longer need to install them as plugin's. I have expressed in the pass if Tom adds these tidbits in I feel better someone in the know and owns and supports the product did it right (linux is not my thing, I add just what I need to get the proper functionality of a NAS).

[trurl]

See the position being missed here is that some don't need and don't want to use anyone elses flavor of VM, and be depended on it, you get hit by a school bus tomorrow and they don't know what to do.

 

My understanding is that you can boot the 64bit Unraid without Xen and install 64bit plugins the same way as you did with 32bit plugins on the 32bit version of unraid.

Look around the forum. People are already converting many of the 32bit plugins to run on 64bit. Whether it is best to go this way or with VMs may depend on specifics. I am currently running 64bit with APCUPSd and Dynamix, mostly because I think UPS and email notifications belong in core unRAID. Depending on how long it takes to get virtualization going I may temporarily add some other plugins but I think a lot of the other stuff belongs in VMs.

 

If we do keep plugins going we need some better package management than everyone doing their own thing like we have had in 32bit unRAID.

 

I think we are repeating ourselves.

[madburg]

I don't disagree with that and that's why the "APPS" i run are on VMs not on unRAID, you said it better certain things belong in core unRAID and if Tom doesn't add (always pledging he does), then we have no choice but to; apcupsd is a perfect example.

[bkastner]

A question: above it was mentioned hardware being passed through. In ESXi doing this means no other VM can have access, does Xen have this same limitation? I assume so, does that mean that the paravirtualized drivers are a near must? Does Windows have these as well under Xen or KVM? If I do passthrough of hardware is there an advantage to running a more capable video card? I've seen talk of virtualizing XBMC for instance, what's the front end for that at the TV? Guess that was more than one question ;-)

 

I am not an expert, but would say that yes, if you are dedicating hardware to a VM you likely can't share it with another VM regardless of which virtualization tech you use (this may not be true 100% of the time, but for a video card for instance it would be).

 

the point of virtualizing XBMC is that there is no front end at the TV - the front end XBMC client is virtualized on your server and you are only running HDMI (and likely IR) to the TV from the backend server. There are likely some TVs you will still need a front end device for (i.e. HDMI is only good for 50 feet unless you extend over CAT5/6, but many of these solutions require 2 CAT cables for HDMI). You also still need to be able to get a remote working with these virtualized solutions as well (hence the IR comment above).

 

Virtualizing UnRAID will likely work for some of your use cases, but not all (at least that is how it looks for me at the moment).

 

For me I have played with a number of end point devices (Pi, Pivos, NUC, Boxee, full PC) and have only really been happy with the Intel NUC & Full PC). I am hoping to consolidate several clients at the server level and minimize the machines floating around my house.

 

[SchoolBusDriver]

See the position being missed here is that some don't need and don't want to use anyone elses flavor of VM, and be depended on it, you get hit by a school bus tomorrow and they don't know what to do.

 

Creating a VM Appliance 101

 

1. Create a Ubuntu image file (img, cow, vdmk, vhd, etc.)

 

2. Install Ubuntu.

 

3. Install App(s) (Example: "apt-get install owncloud.")

 

4. Poweroff VM.

 

5. Upload VM Appliance image and simple VM cfg to your google drive or dropbox.

 

6. Create thread with overview of which OS you used, which Apps you installed in the Appliance, instructions on how to access the GUI / WebGUI and links to the VM Appliance in google drive / dropbox.

 

7. If you really wanted to put it over the top... Post any number of 1,000+ links to the Apps website, forum, or blogs for how to use / install / configure that App (or Apps) in the same OS you used if the user needs additional help / info.

 

Phew! That is some serious complicated stuff there.

 

I'm sure if I get hit by my Bus... There are 500+ plus people on here who can easily do Step 1 - 7 for someone but less than 10 who can write / maintain a plugin.

 

If someone wants to virtualize they have turnkey hypervisors for that today, right now.

 

Name me one company you know, post one white paper or any link on the web where ANY IT Consultant / Professional, Red Hat, Oracle, EMC, NetApp, etc. even HINTS that you should pass through drives / data (that you value / care about the integrity of it) to a VM (home users like yourself do not count).

 

I will save you the trouble... YOU WON'T. In fact, you will find whitepapers, IT Professionals / Consultants who have a whole lot more knowledge / experience / money on the line than we do... Tell you infactly NOT TO DO IT. If you were an IT Professional / Consultant and suggested it... you would be laughed out of the room. If you actually did it, you would be fired.

 

But to have to wait for someone to create a plugin for unRAID and it may or not work well and as easy as other turn key hypervisor already out there is questionable. I agree with dalben comment and believe in the unRAID I purchase for what it does being a good NAS which spins down drives.

 

You know this is OPTIONAL right? If you do not want to run a Type 1 Hypervisor on unRAID... DON'T.

 

For those who don't have the money/interest/knowledge to do something like the above to offload apps like sab, sickbeard, via VMs etc.. they want it to run on unRAID as a plugin, and with that, the hope was with a x64 bit version from Tom, it would be what is require to run those type of plugin and not run into memory issues as they currently are it the 32 bit version.

 

Can you explain to me why they will not be able to do that?

 

I'm running 64-Bit plugins on my unRAID 6.0 and so are several other people on here.

 

If your complaint is a lack of plugins... Why don't you PAY the Plugin Developers who write 2,000+ lines of code and maintain it for FREE. I suspect if you threw some money to those guys, there would be more incentive for them to maintain what they have now and even produce new ones.

 

What it sound and looks like at this moment is, yeah here is the 64 bit version, but don't run plugins, run them with a VM through unRAID with no solid easy way to do so (as it stands).

 

Maybe my eyes are lying to me but I just double checked... My plugins are running and they are even 64-Bit. I even double checked the forum and I see many of the popular ones have been updated / converted to 64-Bit within days of the first beta release.

 

Am I the only one who is running 64-Bit Plugins or is anyone else downloading and installing them?

 

So I ask why? I have my chosen hypervisor today which I can spin up an VM OS i want with a few clicks and just want a 64 bit unRAID, to be able to say run cache-dirs on ALL the content and preclear drives at the same time, run parity check without things boming out.

 

1. Many people cannot use ESXi with their hardware even if it does VT-D and IOMMU and only after loading ESXi 5.0, 5.1 or 5.5 will they be able to determine that.

 

2. It's CRIPPLEWARE if you want Virtual Machine 10 functions / features.

 

3. Many people do not want to pay $700+ dollars to uncripple it.

 

Why should we suffer when you can go on doing what you want and none of the Xen / KVM makes one difference to your setup, your ability to run it on ESXi, stability or performance? Tom turned on the VMWare Guest support and the PV drivers which you were happy to get... Why can't we enjoy the same?

 

You can continue as you are. You can install unRAID 64-Bit and do exactly what you are doing now in unRAID 5.0 if you want and would never know / notice that Xen or KVM is turned on.

 

TO the point above if I only have 4-8 gb of ram I just want a few thing say like sab/sickbeard/couch to run on unRAID. Why cant have person run it as a plugin on a 64 bit version, why would they have to run a VM and allocate it memory to run those three apps, that just doesn't make sense.

 

My Xen, XBMC (playing a movie), unRAID, mysql and 10+ apps on my "custom" unRAID... Takes up less than 1 GB of ram.

 

My VM with SABnzb, Couchpotato, Sickbeard, etc. has 1 CPU assigned (which I share with 20 VMs if I wanted) and takes up less than 512MB of RAM.

 

I still have 2.5 GB of RAM just sitting there not being used. Adding more RAM or CPUs to the HOST (unRAID) doesn't make data write any faster to my unRAID drives or my parity check run faster.

 

P.S. I am not saying to ripe XEN out, if its a small footprint and makes advanced linux users happy to pay with, sure they can knock themselves out and play.

 

What is your point then?

 

Plugins will exist as they do today and Tom has confirmed it several times. Do you doubt him?

[RockDawg]

Geez SchoolBus/Grumpy! You act like he cut down your mother or something. What is your vested interest in all this? I thought you already had your system set up.

 

You should really calm down a bit and allow others their opinion.  I don't use plugins myself, but I could see where some fear all this virtualization may cause developers to stop working on plugins. That's a fair concern even if it won't affect your little world.

 

Sent from my SM-N900V using Tapatalk

 

 

[PeterB]

vif = ['mac=00:16:3e:01:01:01,bridge=xenbr0'] # <--- The MAC Address and bridge name for the VM.

 

The rest of your example config is relatively obvious.  However, what I'm not clear on is whether vif refers to a logical interface, with a logical mac, or is this the real mac of the physical interface that the VM should be using?  I'm guessing that it's the second, in which case, those of us with multiple ethernet interfaces can split the network traffic - unRAID base via one port and VMs via another port?

[PeterB]

the point of virtualizing XBMC is that there is no front end at the TV - the front end XBMC client is virtualized on your server and you are only running HDMI (and likely IR) to the TV from the backend server. There are likely some TVs you will still need a front end device for (i.e. HDMI is only good for 50 feet unless you extend over CAT5/6, but many of these solutions require 2 CAT cables for HDMI). You also still need to be able to get a remote working with these virtualized solutions as well (hence the IR comment above).

 

I guess that would be a real option for me - I would only need a 3 metre cable from the unRAID box to one of my TVs - would it be possible to virtualise OpenELEC and would my cec-usb interface still work?

[BLKMGK]

I don't imagine plugin developers being scared away by virtualization, that makes no sense. However some of the harder to maintain plugins that maybe work more easily In a standard Linux VM might not get the same love. Makes sense to me anyway but since all of that is volunteer their motivations are their own.

 

Now I better understand what folks do with virtualized XBMC! I have a pair of Haswell NUC, one i3 OpenElec and an i5 full Mint14. I'll stick with them but was wondering what magic I was missing to distribute a video signal lol! An HDMI extender pushing HDMI over cat5 might work but my server really is tucked away from all of my TVs. Thanks for explaining! One day I might try PXE booting them maybe.

 

Grumpy - not everyone is keeping track of developments as closely as others. They see lots of excitement over virtualization and become worried that the product they've grown used to is suddenly changing. Many of us that are excited about virtualization realize that's not the case and that they will be able to continue to use the system as they have before. If folks are nervous and ask questions out of ignorance not having kept up on this we really just need to reassure them and try not to get too frustrated that they aren't reading up as much as we might like. In the end I think this will work great for everyone but in the meantime we may need to reassure folks that the rug isn't being pulled out from under them!

 

Peter - yes it should be possible to split traffic. I'm not sure how Xen and KVM compare to ESXi regarding networking but ESXi can setup virtual networks and do some pretty crazy stuff (I've considered virtualizing my firewall in a VM on a different "network"). Software routers and all sorts of things can be done, I'll be surprised if we can't do at least some of this and more as things mature - makes my head spin!

 

 

Sent from my iPad using Tapatalk

[BLKMGK]

Peter virtualizing OpenElec should be possible although I'm not clear on some of the paravirtualizing drivers Grumpy mentioned before so it might not be as clear cut as on ESXi. The CEC would require a USB interface passed in maybe and possibly even an entire USB hub. I'd stay tuned and to make plans just yet, maybe Grumpy. Will share his XBMC VM ;-)

 

 

Sent from my iPad using Tapatalk

[coppit]

Name me one company you know, post one white paper or any link on the web where ANY IT Consultant / Professional, Red Hat, Oracle, EMC, NetApp, etc. even HINTS that you should pass through drives / data (that you value / care about the integrity of it) to a VM (home users like yourself do not count).

 

I'd like to learn more about this, but Google fails me. Would you be so kind as to point me to a white paper or two that describes the technical rationale for not passing through disks to a VM? Or if you are inclined, share one or two of those technical reasons?

 

This blog post from the Hyper-V team gives instructions on how to use passthrough disks, and doesn't warn about it: http://blogs.technet.com/b/askcore/archive/2008/10/24/configuring-pass-through-disks-in-hyper-v.aspx

 

I found this white paper: http://blogs.msdn.com/b/taylorb/archive/2010/02/26/hyper-v-virtual-hard-disk-vhd-performance-white-paper.aspx

 

It says pros of pass-through disks are:

 

• Fastest performance
  
• Simplest storage path because file system on host is not involved.
  
• Better alignment under SAN.
  
• For shared storage based pass-through, no need to mount the file system on host and that may speed up VM live migration.
  
• Lower CPU utilization
  
• Support very large disks
  

 

And cons are:

 

• VM snapshot cannot be taken
  
• Disk is being used exclusively and directly by a single virtual machine.
  
• Pass-through disks cannot be backed up by the Hyper-V VSS writer and any backup program that uses the Hyper-V VSS writer.
  

 

Those seem irrelevant for a virtualized unraid scenario.

 

Don't get me wrong... I'm planning on using unraid as a dom0 Xen host. But I'd like to learn more about the problems of virtualizing unraid. I might go that route initially, since it's possible that it might be a while before unraid 6.0 hits final.

[BLKMGK]

I can't point to any papers but I can say I've gotten some very strange looks mentioning that I pass through controllers when I've discussed this with folks who know more about ESXi than I do! Passthrough is apparently not done by many was the impression i got - distinctly. The HardOCP virtualization and storage forums may be good places to read up on some of this, I've learned some reading in there - no Tapatalk tho dammit.

 

Do note that on ESXi and probably others you don't just pass A drive through, the entire subsystem like a SAS card or USB hub gets passed - it's fairly drastic. No snapshots sux! My .02...

 

 

Sent from my iPad using Tapatalk

[SchoolBusDriver]

You should really calm down a bit and allow others their opinion.

He stated his opinion that we cannot install unRAID 6.0 and use plugins. I stated my opinion and the clear evidence that we can use plugins or VMs if we so desire.

 

I could see where some fear all this virtualization may cause developers to stop working on plugins. That's a fair concern even if it won't affect your little world.

 

That is probably what will happen.

 

Based on several threads regarding plugins, the going rate on plugin develop was a lot of begging / pleading plus an additional $20. I suspect that number is going to continue to climb since only a handful of people actually know how / want to write / maintain plugins.

 

Would you spend your free time for the next several weeks / months to write 500 / 2000+ code for a plugin that might not ever be stable and maintain / update it through the entire alpha, beta and release candidate process for unRAID 6.0 without getting paid?

 

Especially when you can install a VM Appliance that many of us will make / share here in less than 5 minutes? Or create your own VM Appliance in 30 minutes or less following 7 easy steps I listed above using any of the 1,000+ blogs, forum posts, wikis, etc. which walk you through the process?

 

Installing Ubuntu in a VM and typing "apt-get install owncloud" or selecting owncloud from the Package Manger GUI (if you load a desktop) is a hell of a lot easier than learning how to write plugins that are probably going to break with every alpha, beta and release candidate and even when 6.0 is final... will still be flaky at best.

 

This 8-track player I have in my car is awesome. I called several recording studios and left them a message begging / pleading for them to release a few of the new albums I want on 8-track. I told them to page / beep when they made up their mind. Problem is, I can't find a company where I get can get beeper or service. Even if I did... I can't find a payphone to call them anyway. This 8-Track player might have to go so I can avoid all the hassles. Cassette Tapes... here I come!

[coppit]

the entire subsystem like a SAS card or USB hub gets passed - it's fairly drastic.

 

Actually, that was my plan.

 

I wonder if one of the reasons we don't hear about it and get odd looks is that unraid is a software raid, and enterprises would opt for NetApp or Isilon over anything like that.

 

Just as a counter-argument, I would get lots of blowback at work for installing things into the Xen dom0. In fact, the Xen Server docs advise that you treat dom0 like an appliance. They don't even support simple RPM-based upgrades any more. You have to reinstall dom0 from the ISO to upgrade. So mixing up unraid and Xen in dom0 could send enterprise folks into conniptions too.