The (un)official unRAID 6.x plugin discussion thread

[sparklyballs]

Whar happened to all the talk about not needing vt-d support with 64-bit?

 

Sent from Moto G using Tapatalk

 

 

my understanding is that you don't need vt-d support to run VM's if you're not going to pass through any hardware to the VM's.

 

if unraid is the base OS then you won't need to pass through controller cards etc....

if however you want to pass things like graphics cards or tv cards to VM's then you need vt-d support.

[nicinabox]

Then to upgrade any package would be done via a repo hosted by the community, as previously mentioned if it's on Arch User Repo (AUR) then we can easily include it our community repo. Example (if VM was in Arch): pacman -S sabnzbd would install sabnzbd, pacman -Syu would upgrade anything that needed it from our repo. Simple stuff!

 

It's probably worth pointing out that boiler and package-virtualization aren't mutually exclusive. Boiler is agnostic to what your package is and it provides a standard distribution mechanism while doing packaging on the host OS.

 

You could in theory use the boiler registry to distribute packages designed for the virtualized environment. The registry is basically akin to a linux repo. Every package system needs some origin source from which to pull packages [1].

 

Boiler takes the core package and maps the files work for unRAID [2]. The same core package could be remapped for any other OS with a simple configuration, and there's still only 1 package to maintain. That cuts down on potential bugs and maintenance overhead that would be incurred by multiplying platforms and architectures. It's a similar to a build-once-run-anywhere approach.

 

THEN AGAIN... maybe maintaining a separate repo for virtualized environments is the right way. Some of this just needs to have some prototypes built to see what really works.

 

Does that make any sense at all?

 

[1] I designed the registry to be maintainable by the community, without being burdensome. Anyone can add it to it without intervention.

[2] unRAID has unique challenges with the RAMfs, so boiler maps stuff automatically to persist configurations, do updates, put bin's in  your PATH, etc.

[dirtysanchez]

Whar happened to all the talk about not needing vt-d support with 64-bit?

 

Sent from Moto G using Tapatalk

 

Since unRaid will be the host you won't need VT-d support unless you want some device passed-through to a VM. VT-x is all you will need, and most all CPU's (at least since Core2) support it.

 

With unRaid as the host there is no longer the need to pass the SATA controllers through to unRaid, hence VT-d not being needed.

[BLKMGK]

If unRAID is being run AS a VM then VT-d is needed so that drive controllers and USB (license key!) can be passed through to unRAID. If unRAID is the host OS then VT-d isn't strictly needed as hardware to the running VMs is virtual, make sense? Let me lay out what I've done to see if it helps folks understand the possibilities.

 

My current system is running ESXi. UnRAID isn't the host OS currently (ESXi is) and I do occasionally see some weirdness who's source I'm unsure of. SAB, SB, a usenet indexer, Plex, MySQL, and other software also run on this system but not within unRAID. UnRAID is fairly pure with few plugins. SAB, Sick, Maraschino, Couch, and other things run in a single preconfigured slim Linux VM. They have their own disk but also have an NFS mount to unRAID. They don't see the storage disks directly but through this network mount - the package mentioned above  might have them working through a shared directory to the host OS instead but I'm not sure this necessary. Tomatoe vs tomaaato - both work. Truthfully no VMs or software have to have a shared space and can talk via the network just as if they were on disparate machines if you want, ESXi actually has a special software NIC that runs way faster than normal to speed this when talking internally on the same box. :-)

 

In my situation if I changed things unRAID might be the host OS, SAB, Sick, Maraschino, and Couch would still run on a single Linux VM with access to an unRAID NFS mount to save files. Plex might be moved from its Win7 VM to Linux maybe, MySQL would probably stay in its own VM. My indexer VM would stand alone. The VM I currently run for FreeNAS to provide fast disk for VM storage could go away, I'd be using unRAID. I do worry about the speed but we'll cross that bridge later.

 

My hardware is beefy but not over the top IMO. A Xeon 4core hyperthreaded CPU of Sandybridge vintage, 32gig ECC RAM, 3x M1015 SAS cards flashed to act as straight adapters, I can support 24+ drives. I run at least 5 VM at all times and I've yet to max out my server according to my system reports. If I could find a decent Haswell Xeon mobo I'd upgrade as part of my V6 testing just to have it lol.

 

Currently I do see weirdness with unRAID hosted as a VM. It's sometimes very slow to respond to SMB requests from my desktop. Drive spin-up is always going to bite me but this occurs even with everything spinning. SMB speeds also fluctuate, sometimes stop, and generally are a pita but work fine for media serving. My hope is that unRAID acting as host will behave better but I want to test before moving as this is pretty ambitious to try and move everything. Currently I have access to somewhat robust management tools with ESXi, I would want that with whatever solution I end up with.

 

If this sounds daunting then continue to use unRAID as you do now. NOTHING about what we're talking about is going to prevent that if you choose to stay the course and your hardware can remain untouched. Plugins should still be around and you could ignore the virtualization easily. Eventually I suspect many will get curious though if the right environment is built :-)

 

I went this path because I had multiple unRAID servers that I could collapse together and I had a ton of services running on my desktop (and some on other low power machines) which were all in my way. I also wanted to explore the tech because I use it at work and the knowledge transfers. Now I have a single large server, a desktop, and HTPC front-ends plus a small firewall. I may yet move my firewall to a VM. My office is quieter and its cooler too! I can manage everything centrally, spin up play sandboxes for new science projects, and if I screw something up roll back my changes. I don't even need to be at home to be able to do this. I consolidated my hardware and software just as any business moving to "the cloud" would and reaped the same benefits...

 

I hope that helps folks understand the possibilities and potential!

 

 

Sent from my iPad using Tapatalk

[zoggy]

There's a lot of talk here advising users to run apps in a VM and not under unRAID.

 

The main reason why I'm interested in (and currently testing) 6.0 is because cache_dirs crashes my box when it uses too much lowmem. I understand 64-bit essentially removes that problem and I will be able to use most of my 8GB in one unified memory model?

 

OK, but cache_dirs can't run under a VM can it? So running a VM won't make any difference there?

 

I run 3 other apps currently under 5.x -- nzbget, Sick Beard and Transmission. They don't crash the box.

 

nzbget needs to be able to read and write quickly to drives.

So, my question is, if I run it in a VM, how does it write to the array? Does it see array drives as (fast) shared drives? Will that slow down writes? Will running nzbget in a VM mean the writes within the VM's own drive are slower?

I'm already now running nzbget in 6.0, thanks to overbyrn.

 

I came to unRAID about 5 years ago from using Buffalo NAS boxes. The great thing about using those was that when rooted, you had a package manager (ipkg), so if you want to install nzbget, you'd type "ipkg install nzbget". And it worked!

 

I don't see the point of running a VM for a couple of apps, especially if performance will be reduced, but perhaps people can enlighten me here.

 

VM's seem to have a bad rep amongst some people, perhaps because of the old days where things like virtualbox didn't have direct hardware access, and had to use virtual devices and NICs etc; They were therefore very slow compared with 'bare metal'. What we are talking about here is a 'type 1' hypervisor such as KVM or Xen, this changes things dramatically.

 

Type 1 hypervisors are used extensively in the enterprise world and are a proven, reliable technology. ESXi is another example of a type 1 hypervisor, but isn't relevant to our discussions as you cannot 'install it as a package' on top of an existing OS like you can with Xen or KVM.

 

If you want an idea of what performance you can expect from a VM then see my youtube video below. Spolier: It's neglible the difference between a Xen VM with the correct drives installed vs a bare metal installation of the same OS. As for how would this apply to a 'plugin specific VM', well that's easy. We'd use the virtio-9p protocol previously discussed here to give the VM access to the host's disks. Now remember the host is unRAID, therefore that's how you get access from let's say Transmission through to unRAID. The VMs would run on an SSD or cache drive.

 

My plan is, once the beta phase has settled down a bit to get some KVM / Xen guides done and release an image of Arch (or something I don't know yet) that you can download and import yourself. Then to upgrade any package would be done via a repo hosted by the community, as previously mentioned if it's on Arch User Repo (AUR) then we can easily include it our community repo. Example (if VM was in Arch): pacman -S sabnzbd would install sabnzbd, pacman -Syu would upgrade anything that needed it from our repo. Simple stuff!

 

Youtube video showing gaming performance using Xen hypervisor running Windows 8 with VGA passthrough

 

Did you know that the Xbox One utilises this technology as well? It runs 3 OS's concurrently. OS#1 is for the hypervisor only (the host), then OS#2 does gaming (a VM no less), then OS#3 handles dashboard / media duties (another VM). This is how it can switch so quickly between activities etc. If it's good enough for an Xbox and enterprise companies you can bet your arse it's good enough for you and I at home.

 

and god is my xbox one slow at switching apps, crashes daily from memory issues, and overall has a ton of bugs. not sure how much of it is just software (core) or software on the games.. but to me this is a horrible real world example.. im sure over time they will get the kinks out. but for right now xbox one for myself and my friends is far far far from how I'd want my nas to perform.

 

up at work we've gone to using vm to consolidate boxes and for there it just works (we also prob have far more resources than the boxes need). i know it has its purpose but i think its going to be a hard sell to a lot of people until they just use it/see the benefit.

 

personally, id love to see a 2gb unraid setup that has just a clean unraid install & sab+sickbeard, and then a 2gb unraid setup with a vm for sab+sickbeard.

seeing how it affects memory, rar unpacking times, how if it negatively impacts things from speed to setup.

[Thornwood]

Hi there every one,

 

From a distribution stand point, the only plugins that I consistently install is Unmenu, for UPS and e-mail, support Dynamatrix or a flavior there of for automation of email and parity checks health also. (Double email support for the different information both send.)

OpenVPN server as needed

MySQL as needed.

 

Customers don't usually administer there Servers (some do)

I usually Do all the admin stuff,

 

If the Plugins are VM(ed) or not I really don't mind out side of one thing I have sometimes gone into the code and modified the code to suit my needs.

Example Unmenu email has no identification on What server is sending the e-mail.

I modified each server to give me there name so I know witch server is having a problem.

so if having everything in a vm is going to make this kind of modification a pain I would rather have it install like they do now.

this also makes remote repairs allot easer I don't have to be in the VM environment to see the files to be able to fix them,

Now if the VM environment can be shared like the flash drive, I would be better but then I still need to be running the drive not just plugged-in to my windows or Linux usb for a quick modification.

 

So to put some prospective there are people that have UnRaid systems that are being administered by a single admin. they have little or no idea of what is under the hood.

These single admins be it a company or friends of the UnRaid users, need some flexibility to keep things easy and cheap for (self and customers).

Please keep this in mind :-)

 

(I don't believe that all this work should be free, I have asked several developers in the past for things that I would pay for, I also will help and write modifications or new code for some but I usually try to help people for free in this forum, I love Unraid and I try to help when I can and this e-mail is just that trying to place a different prospective on how some people use Unraid.)

Very exited on 64 will get a new server started as soon as possible to run Unraid64.

 

your friend

Thornwood

[BLKMGK]

Thornwood, one of the nice things about VMs is that they are their own OS. You could ssh into the VM as easily as another machine. Code modifications should be as easy as they are now. Think of a VM as a software computer, it's an emulated environment. If you've ever used VMware workstation or player, or virtual box, it's like that only run by a server. You can run anything you want in it including Windows if its setup correctly. Or, just run unRAID the way you do now and don't worry about it :-)

 

 

Sent from my iPad using Tapatalk

[nicinabox]

Can someone explain the benefits of using a virtualized environment on a RAM disk?

[BLKMGK]

 

Can someone explain the benefits of using a virtualized environment on a RAM disk?

 

I'll be honest, I've never seen that done and it would have to be a VERY slim environment! In my experience servers supporting virtual machines store them on disk, and in my case I also store snapshots such that if I make a mistake I can roll back to a known good state. Servers supporting many VMs rely heavily on fast disks and heaps of memory. I would suspect we wouldn't be trying to support too many VMs on an unRAID box though.

 

 

Sent from my iPad using Tapatalk

[Bags]

I guess my question was too vague on hardware VM's. I guess I am trying to figure out with all this discussion about VM's, what does this do to all the little guys such as myself who are running less powerful hardware? Am I going to have to scrap my Mobo,processor, and ram to upgrade costing me several $100's?

[dirtysanchez]

Am I going to have to scrap my Mobo,processor, and ram to upgrade costing me several $100's?

 

Running VM's isn't mandatory, you'd need to upgrade only if you decide you want to run VM's.  Your existing hardware will likely run v6 (assuming your hardware is 64-bit capable) just as good as it runs v5. Even if it didn't you could always stay on v5 as it's not going away.

[jowi]

If it's good enough for an Xbox and enterprise companies you can bet your arse it's good enough for you and I at home.

To you this is a good thing, for me... as a linux novice, i am affraid that if i HAVE to create vm's and set up a technically advanced system with hypervisors and other stuff i dont understand, only to run unraid... i think i will stop using unraid.

 

I want to know, WHAT is the exact benefit for having unraid as a 64 bits environment, with the abilty to run vm's or be one itself? Will my data be protected better? Will my transfers go faster?  In fact, i think if something goes wrong, with the current unraid i think i can manage correcting it, if i have a system that consists of vm's within vm's running vm's... i dont know what the hell i am doing anymore.

 

Point is, i think this 64 bits version with vm support should be considered a professional or business version, not for home use... the power of unraid *was* that even people how did not know linux, could plug in a usb stick and have a solid NAS. With this version, not so much...

 

I'm affraid the simple user will be left behind and unraid becomes something only die hard linux techies can setup and run.

[dirtysanchez]

To you this is a good thing, for me... as a linux novice, i am affraid that if i HAVE to create vm's and set up a technically advanced system with hypervisors and other stuff i dont understand, only to run unraid... i think i will stop using unraid.

 

I want to know, WHAT is the exact benefit for having unraid as a 64 bits environment, with the abilty to run vm's or be one itself? Will my data be protected better? Will my transfers go faster?  In fact, i think if something goes wrong, with the current unraid i think i can manage correcting it, if i have a system that consists of vm's within vm's running vm's... i dont know what the hell i am doing anymore.

 

Point is, i think this 64 bits version with vm support should be considered a professional or business version, not for home use... the power of unraid *was* that even people how did not know linux, could plug in a usb stick and have a solid NAS. With this version, not so much...

 

I'm affraid the simple user will be left behind and unraid becomes something only die hard linux techies can setup and run.

 

The only difference between v5 and v6 is that v6 is 64-bit, allowing you to use more than 4GB of RAM more effectively than v5 did. v6 also has virtualization options enabled in the kernel.

 

Will your data be protected better?  No. Will transfers be faster?  Maybe, maybe not.

 

v6 is just as much for home use as any other version was. Load a vanilla install of v6 to your server and it'll run just like v5 did. It will just work. Want to stay with the plugin model?  Fine. You don't HAVE to virtualize anything on your server unless you want to.

 

"People who did not know Linux could plug in a USB stick and have a solid NAS" still can. Nothing has changed in that respect. The "simple user" can still run unRaid the "simple" way. v6 just gives more advanced users the ability to do more advanced things if they so choose.

[jowi]

Want to stay with the plugin model?  Fine. You don't HAVE to virtualize anything on your server unless you want to.

Ok, i did not know that, everybody is hyping the vm model now and talking about how we should and must now create vm's if we want to run a plugin.

Would be nice if the discussion would also cover us 'simple' users and our needs.

[dirtysanchez]

Want to stay with the plugin model?  Fine. You don't HAVE to virtualize anything on your server unless you want to.

Ok, i did not know that, everybody is hyping the vm model now and talking about how we should and must now create vm's if we want to run a plugin.

Would be nice if the discussion would also cover us 'simple' users and our needs.

 

Plugins will still be around and you can continue to use plugins if you wish. In fact, a number of the most popular plugins have already been converted to 64-bit and posted to the forums.

 

Most advanced users would prefer to get rid of plugins on their systems and run those apps in a VM instead because it provides better stability and keeps the core unRaid "clean", but that is by no means mandatory with v6.  It's just a personal preference of some of the more advanced users.

[nicinabox]

Most advanced users would prefer to get rid of plugins on their systems and run those apps in a VM instead because it provides better stability and keeps the core unRaid "clean"...

 

FWIW this has been a very repeated mantra with no actual tests or benchmarks. Lots of scare quotes though.

[sparklyballs]

Most advanced users would prefer to get rid of plugins on their systems and run those apps in a VM instead because it provides better stability and keeps the core unRaid "clean"...

 

FWIW this has been a very repeated mantra with no actual tests or benchmarks. Lots of scare quotes though.

 

Highly unscientific as it may be but on the same hardware i had nothing but problems (memory hogging causing unraid gui to crash) with sab running as a plugin on unraid which were alleviated to a vast extent by moving sab to a vm. after a while though i completely dumped sab and now use nzbget.

[nicinabox]

Most advanced users would prefer to get rid of plugins on their systems and run those apps in a VM instead because it provides better stability and keeps the core unRaid "clean"...

 

FWIW this has been a very repeated mantra with no actual tests or benchmarks. Lots of scare quotes though.

 

Highly unscientific as it may be but on the same hardware i had nothing but problems (memory hogging causing unraid gui to crash) with sab running as a plugin on unraid which were alleviated to a vast extent by moving sab to a vm. after a while though i completely dumped sab and now use nzbget.

 

These kinds of experiences are extremely useful. Is there any way we could compile real data about the issues (process/memory usage, installed packages, logs, etc)? Unfortunately, on it's own it's highly anecdotal and can lead to misdiagnosis.

[sparklyballs]

Most advanced users would prefer to get rid of plugins on their systems and run those apps in a VM instead because it provides better stability and keeps the core unRaid "clean"...

 

FWIW this has been a very repeated mantra with no actual tests or benchmarks. Lots of scare quotes though.

 

Highly unscientific as it may be but on the same hardware i had nothing but problems (memory hogging causing unraid gui to crash) with sab running as a plugin on unraid which were alleviated to a vast extent by moving sab to a vm. after a while though i completely dumped sab and now use nzbget.

 

These kinds of experiences are extremely useful. Is there any way we could compile real data about the issues (process/memory usage, installed packages, logs, etc)? Unfortunately, on it's own it's highly anecdotal and can lead to misdiagnosis.

 

Find someone with two similar,  or better still identical systems to do a side by side comparison. but sab is known to be a major memory hogger in of itself and would be my prime candidate for shifting to a vm if i were still using it.

[nicinabox]

Most advanced users would prefer to get rid of plugins on their systems and run those apps in a VM instead because it provides better stability and keeps the core unRaid "clean"...

 

FWIW this has been a very repeated mantra with no actual tests or benchmarks. Lots of scare quotes though.

 

Highly unscientific as it may be but on the same hardware i had nothing but problems (memory hogging causing unraid gui to crash) with sab running as a plugin on unraid which were alleviated to a vast extent by moving sab to a vm. after a while though i completely dumped sab and now use nzbget.

 

These kinds of experiences are extremely useful. Is there any way we could compile real data about the issues (process/memory usage, installed packages, logs, etc)? Unfortunately, on it's own it's highly anecdotal and can lead to misdiagnosis.

 

Find someone with two similar,  or better still identical systems to do a side by side comparison. but sab is known to be a major memory hogger in of itself and would be my prime candidate for shifting to a vm if i were still using it.

 

Forgive my ignorance here... How does a VM prevent a memory issue in a running process? Wouldn't you end up with the same problem in that environment? Won't the VM need to draw resources from the host system to accommodate ballooning processes?

 

(not trolling here, I really would like some technical explanations)

[sparklyballs]

Most advanced users would prefer to get rid of plugins on their systems and run those apps in a VM instead because it provides better stability and keeps the core unRaid "clean"...

 

FWIW this has been a very repeated mantra with no actual tests or benchmarks. Lots of scare quotes though.

 

Highly unscientific as it may be but on the same hardware i had nothing but problems (memory hogging causing unraid gui to crash) with sab running as a plugin on unraid which were alleviated to a vast extent by moving sab to a vm. after a while though i completely dumped sab and now use nzbget.

 

These kinds of experiences are extremely useful. Is there any way we could compile real data about the issues (process/memory usage, installed packages, logs, etc)? Unfortunately, on it's own it's highly anecdotal and can lead to misdiagnosis.

 

Find someone with two similar,  or better still identical systems to do a side by side comparison. but sab is known to be a major memory hogger in of itself and would be my prime candidate for shifting to a vm if i were still using it.

 

Forgive my ignorance here... How does a VM prevent a memory issue in a running process? Wouldn't you end up with the same problem in that environment? Won't the VM need to draw resources from the host system to accommodate ballooning processes?

 

(not trolling here, I really would like some technical explanations)

 

running a vm if you run into a memory issue, it will drag down the vm only....

[nicinabox]

Most advanced users would prefer to get rid of plugins on their systems and run those apps in a VM instead because it provides better stability and keeps the core unRaid "clean"...

 

FWIW this has been a very repeated mantra with no actual tests or benchmarks. Lots of scare quotes though.

 

Highly unscientific as it may be but on the same hardware i had nothing but problems (memory hogging causing unraid gui to crash) with sab running as a plugin on unraid which were alleviated to a vast extent by moving sab to a vm. after a while though i completely dumped sab and now use nzbget.

 

These kinds of experiences are extremely useful. Is there any way we could compile real data about the issues (process/memory usage, installed packages, logs, etc)? Unfortunately, on it's own it's highly anecdotal and can lead to misdiagnosis.

 

Find someone with two similar,  or better still identical systems to do a side by side comparison. but sab is known to be a major memory hogger in of itself and would be my prime candidate for shifting to a vm if i were still using it.

 

Forgive my ignorance here... How does a VM prevent a memory issue in a running process? Wouldn't you end up with the same problem in that environment? Won't the VM need to draw resources from the host system to accommodate ballooning processes?

 

(not trolling here, I really would like some technical explanations)

 

running a vm if you run into a memory issue, it will drag down the vm only....

 

And that would ultimately cause it to fail, right? So why isn't the problem being fixed in the application causing the problem?

[sparklyballs]

Most advanced users would prefer to get rid of plugins on their systems and run those apps in a VM instead because it provides better stability and keeps the core unRaid "clean"...

 

FWIW this has been a very repeated mantra with no actual tests or benchmarks. Lots of scare quotes though.

 

Highly unscientific as it may be but on the same hardware i had nothing but problems (memory hogging causing unraid gui to crash) with sab running as a plugin on unraid which were alleviated to a vast extent by moving sab to a vm. after a while though i completely dumped sab and now use nzbget.

 

These kinds of experiences are extremely useful. Is there any way we could compile real data about the issues (process/memory usage, installed packages, logs, etc)? Unfortunately, on it's own it's highly anecdotal and can lead to misdiagnosis.

 

Find someone with two similar,  or better still identical systems to do a side by side comparison. but sab is known to be a major memory hogger in of itself and would be my prime candidate for shifting to a vm if i were still using it.

 

Forgive my ignorance here... How does a VM prevent a memory issue in a running process? Wouldn't you end up with the same problem in that environment? Won't the VM need to draw resources from the host system to accommodate ballooning processes?

 

(not trolling here, I really would like some technical explanations)

 

running a vm if you run into a memory issue, it will drag down the vm only....

 

And that would ultimately cause it to fail, right? So why isn't the problem being fixed in the application causing the problem?

 

 

cos sabnzbd looks pretty but is ultimately a pile of shit....

[dirtysanchez]

Most advanced users would prefer to get rid of plugins on their systems and run those apps in a VM instead because it provides better stability and keeps the core unRaid "clean"...

 

FWIW this has been a very repeated mantra with no actual tests or benchmarks. Lots of scare quotes though.

 

While there are no statistics or hard data I can point to, all one has to do is take a look at the forums. The number of support threads involving GUI crashes or dependency issues from users not running any plugins at all is virtually non-existent. Users running a large number of plugins, OTOH, there's untold numbers of threads.

 

For the record, I've been lucky. I run a number of plugins on my server and haven't had a single issue. It's been rock solid stable. But I also know there are lots of users who have had nothing but issues with plugins.

[BLKMGK]

When your computer runs out of memory it runs out and bad things happen or it swaps etc. etc. - yes? A VM is a virtual computer, if I tell it that it can have say 4gig of memory but its a pig and wants more the VM can limit it or if there's spare host RAM give it to the VM. In the ESXi world this increase in RAM is called ballooning. One of the really nice things is that if something goes REALLY bad and crashes hard the virtual computer should be the only thing effected and not bring down the entire computer. This takes process isolation to a whole new level! Software installed for one thing won't effect other things. Need something that works best in Windows? Load a Windows VM and run it, likewise different flavors of Linux. Done right this allows you to best leverage a single hardware platform and not have a bunch of memory or CPU sit idle - this is mostly important In a corp. environment but has proven true in my home where multiple computers were collapsed into one really capable unit. :-)

 

 

Sent from my iPad using Tapatalk