jonp Posted April 28, 2015 Share Posted April 28, 2015 Re: Plugins - Our official policy is that we highly discourage their use. There are two main reasons for this: 1. Plugins are inherently a huge security risk. They are installed with 'root' privilege. Many plugins install 3rd party packages from who-knows-where containing who-knows-what. 2. Plugins will always suffer from problems with incompatible dependencies. unRaid OS is not a distro like say Arch or debian. One way to mitigate risks with plugins is to work with certified plugins only. I know this will give an addiional burden but at the end of the day it will give ensurance to both LT and the end-user that a particular plugin is suitable and compatible. Certified plugins should be listed (downloadable) from a managed site, this will make it easy to find them and may give the oppertunity to distinguish between plugins and dockers, read: give preferential advice. I understand that all focus is now on Docker and VM, but I believe there is still place and reason to have plugins. Not all plugins require third party software. The statement highly discourage doesn't sound like an invitation to continue on plugins, while they are still very useful. Knowing how much time and effort I've put in developping plugins (and the plugin manager), your announcement is 'bitter sweet'. Your work is an exception to the rule for the most part because your work falls in the category of webgui enhancements. Having certified plugins is something we have talked about doing and both python and Perl are examples of packages that could fall into that category. Quote Link to comment
bonienl Posted April 28, 2015 Share Posted April 28, 2015 Re: Plugins - Our official policy is that we highly discourage their use. There are two main reasons for this: 1. Plugins are inherently a huge security risk. They are installed with 'root' privilege. Many plugins install 3rd party packages from who-knows-where containing who-knows-what. 2. Plugins will always suffer from problems with incompatible dependencies. unRaid OS is not a distro like say Arch or debian. One way to mitigate risks with plugins is to work with certified plugins only. I know this will give an addiional burden but at the end of the day it will give ensurance to both LT and the end-user that a particular plugin is suitable and compatible. Certified plugins should be listed (downloadable) from a managed site, this will make it easy to find them and may give the oppertunity to distinguish between plugins and dockers, read: give preferential advice. I understand that all focus is now on Docker and VM, but I believe there is still place and reason to have plugins. Not all plugins require third party software. The statement highly discourage doesn't sound like an invitation to continue on plugins, while they are still very useful. Knowing how much time and effort I've put in developping plugins (and the plugin manager), your announcement is 'bitter sweet'. Your work is an exception to the rule for the most part because your work falls in the category of webgui enhancements. Having certified plugins is something we have talked about doing and both python and Perl are examples of packages that could fall into that category. Sure, but the statement given is rather black and white, and people may interprete the message as "never use plugins because they are unsafe and pose problems". A little nuance would do. Certification definitely will help the stability of your product in the long run, that may also be true for dockers ! Quote Link to comment
sparklyballs Posted April 28, 2015 Share Posted April 28, 2015 Re: Plugins - Our official policy is that we highly discourage their use. There are two main reasons for this: 1. Plugins are inherently a huge security risk. They are installed with 'root' privilege. Many plugins install 3rd party packages from who-knows-where containing who-knows-what. 2. Plugins will always suffer from problems with incompatible dependencies. unRaid OS is not a distro like say Arch or debian. One way to mitigate risks with plugins is to work with certified plugins only. I know this will give an addiional burden but at the end of the day it will give ensurance to both LT and the end-user that a particular plugin is suitable and compatible. Certified plugins should be listed (downloadable) from a managed site, this will make it easy to find them and may give the oppertunity to distinguish between plugins and dockers, read: give preferential advice. I understand that all focus is now on Docker and VM, but I believe there is still place and reason to have plugins. Not all plugins require third party software. The statement highly discourage doesn't sound like an invitation to continue on plugins, while they are still very useful. Knowing how much time and effort I've put in developping plugins (and the plugin manager), your announcement is 'bitter sweet'. Your work is an exception to the rule for the most part because your work falls in the category of webgui enhancements. Having certified plugins is something we have talked about doing and both python and Perl are examples of packages that could fall into that category. Sure, but the statement given is rather black and white, and people may interprete the message as "never use plugins because they are unsafe and pose problems". A little nuance would do. Certification definitely will help the stability of your product in the long run, that may also be true for dockers ! certification for dockers...... and an unraid base image, doesn't need to be in the bzroot or anything, just on dockerhub.. with a few choice extras over and above a standard base image. Quote Link to comment
Squid Posted April 28, 2015 Share Posted April 28, 2015 Re: Plugins - Our official policy is that we highly discourage their use. There are two main reasons for this: 1. Plugins are inherently a huge security risk. They are installed with 'root' privilege. Many plugins install 3rd party packages from who-knows-where containing who-knows-what. 2. Plugins will always suffer from problems with incompatible dependencies. unRaid OS is not a distro like say Arch or debian. We have spent a great deal of effort refining replacements to the "plugin" system: docker containers and virtual machines. Using these technologies our goal is to reduce the core OS even further. I can see 'plugins' still being useful for webGui enhancement (where all the source is visible), but not for adding executables. I dislike the terminology of "highly discourage", and still being only useful for webGUI enhancement. There are a number of plugins that are impractical to program as a docker container (Docker Search & XML conversion) that could not be classified as a GUI enhancement, and yet are only practical as a plugin. (Although I try and avoid dependencies like the plague) While I completely understand trying to get away from application plugins and move those to docker containers (and have been encouraging the community as a whole to stop using application plugins), limiting ourselves to GUI enhancements is almost shooting yourself in the foot. Quote Link to comment
jonp Posted April 28, 2015 Share Posted April 28, 2015 Re: Plugins - Our official policy is that we highly discourage their use. There are two main reasons for this: 1. Plugins are inherently a huge security risk. They are installed with 'root' privilege. Many plugins install 3rd party packages from who-knows-where containing who-knows-what. 2. Plugins will always suffer from problems with incompatible dependencies. unRaid OS is not a distro like say Arch or debian. We have spent a great deal of effort refining replacements to the "plugin" system: docker containers and virtual machines. Using these technologies our goal is to reduce the core OS even further. I can see 'plugins' still being useful for webGui enhancement (where all the source is visible), but not for adding executables. I dislike the terminology of "highly discourage", and still being only useful for webGUI enhancement. There are a number of plugins that are impractical to program as a docker container (Docker Search & XML conversion) that could not be classified as a GUI enhancement, and yet are only practical as a plugin. (Although I try and avoid dependencies like the plague) While I completely understand trying to get away from application plugins and move those to docker containers (and have been encouraging the community as a whole to stop using application plugins), limiting ourselves to GUI enhancements is almost shooting yourself in the foot. I would classify your plugins as GUI enhancements as well. They enhance the user experience and simplify things. They do not install 3rd party software packages. The primary point we are making here is that the issues are of the highest concern when application packages are installed through plugins. There is no mechanism for users to control the exposure plugins get to data on the system. This contrasts directly with containers and VMs which can have their access restricted to only specific folder paths and even read only access at that should the user desire. Same goes for system resource allocation. We can more easily limit applications to CPU, disk, network, and memory resources with virtualization than with plugins. Before virtualization technologies like Docker and VMs, there was no alternative. With unraid 6, we have those alternatives and they happen to solve other key problems as well for application maintainers. I realize some may think this is a heavy handed approach, but you really need to reread Tom's post about what plugins can really do. Quote Link to comment
bonienl Posted April 28, 2015 Share Posted April 28, 2015 But it's an illusion to think that "non-app" plugins never require additional packages. E.g. the Dynamix Stats plugin uses the package "sysstats", eventhough it is considered a GUI enhancement. Quote Link to comment
gundamguy Posted April 28, 2015 Share Posted April 28, 2015 Many plugins install 3rd party packages from who-knows-where containing who-knows-what. As someone who hasn't spent much time messing with docker yet, is that really solved by Docker Containers? I've seen plenty of threads in the docker forums where people are installing dockers from who-knows-where containing who-knows-what? I might be thinking about this wrong, becuase again I admit that I've not spent much time messing with Docker (Plugin's were the only option when I started with unRAID, and they just work) but I see Containers as black boxes (made by someone else using packages from who-knows-where containing who-knows-what) that we are installing into our system and having to trust that the black box doesn't also happen to contain something we don't want. I do agree that the potential for abuse is more limited by the nature of containers, but I don't think that means it's entirely gone. Quote Link to comment
jonp Posted April 28, 2015 Share Posted April 28, 2015 But it's an illusion to think that "non-app" plugins never require additional packages. E.g. the Dynamix Stats plugin uses the package "sysstats", eventhough it is considered a GUI enhancement. Right, which isn't something we would probably advocate for folks to run on a production system because it hasn't been tested. If you recall, there were issues with the system stats plugin that was affecting notifications at one point. Adding packages changes the environment and therefore is not tested by us at Lime Tech. For those willing to take the risk to help beta test community work, that's fine, but we can't encourage users to install 3rd party packages with root access to the system if we ourselves are not using them. There are a number of cases documented now where plugins were the cause for system instability. We saw this with Cache Dirs, I've seen it with system stats. I've seen it with SNAP. Bottom line, what "works" and what's "stable and supported" are two different things. As far as the other comment about docker installing random packages, please see my other post. Docker at least let's the user control the level of access a container had to a system. Same goes for VMs. Plugins that enhance the user experience of unRAID are considered for inclusion into the OS itself. This includes things like Dynamix, APCUPSd, and the enhancements to DockerMan that gfjardim helped with. We have plans to incorporate others as well, but let's not mince words, until a plugin is merged into the base OS, it is considered a proof of concept and beta code at that. Now, if a plugin is deemed useful, valuable, and is fitting to merge into unRAID OS, we will look at all it does and requires from a dependency standpoint and if deemed suitable, we will officially support it. Quote Link to comment
gundamguy Posted April 28, 2015 Share Posted April 28, 2015 There are a number of cases documented now where plugins were the cause for system instability. We saw this with Cache Dirs, I've seen it with system stats. I've seen it with SNAP. Bottom line, what "works" and what's "stable and supported" are two different things. Docker is a relatively new thing when it comes to unRAID, and it's has it's host of road blocks and setbacks. Currently there is an unresolved issue in the defects report forum about system instability that results from docker. As far as the other comment about docker installing random packages, please see my other post. Docker at least let's the user control the level of access a container had to a system. Same goes for VMs. Now don't get me wrong building containers around programs is an improvement. I just don't want people to have a false sense of security and think that containers are impervious to malicious code or that Docker has had no security concerns. Overall I think Docker is a great thing, I just don't like the amount of hype it's getting, I am concerned that it's getting over hyped. Quote Link to comment
jonp Posted April 28, 2015 Share Posted April 28, 2015 There are a number of cases documented now where plugins were the cause for system instability. We saw this with Cache Dirs, I've seen it with system stats. I've seen it with SNAP. Bottom line, what "works" and what's "stable and supported" are two different things. Docker is a relatively new thing when it comes to unRAID, and it's has it's host of road blocks and setbacks. Currently there is an unresolved issue in the defects report forum about system instability that results from docker. This is the first time since the first inclusion of Docker that any host instability has occurred. Let's not grab the pitchforks just yet. We are actively investigating the issue and testing to see if we can recreate and if Docker 1.6 resolves the issue. As far as road blocks and setbacks, to what are you referring exactly? And keep in mind, the issues with host stability that you are referring to means Docker now has 1 bug that has proven to impact host stability. Do you have any idea how many bugs have existed in plugins that have affected host stability / functionality? As far as the other comment about docker installing random packages, please see my other post. Docker at least let's the user control the level of access a container had to a system. Same goes for VMs. Now don't get me wrong building containers around programs is an improvement. I just don't want people to have a false sense of security and think that containers are impervious to malicious code or that Docker has had no security concerns. We never said that containers were impervious to malicious code. If someone downloaded a container, gave it full access to /mnt/user, and then the container executed a script to remove data, yes, that could happen. The difference is that the user at least has a way to control that universally above and beyond what the container wants to do. I could run the container in an isolate environment and only give it read-only access to the system, thereby preventing this from occurring. With plugins, there is ZERO controls in place for that. The user has no option but to either trust a plugin completely, or not at all. This makes Docker 100x better than plugins from a testing and control standponit. The same applies to VMs. Overall I think Docker is a great thing, I just don't like the amount of hype it's getting, I am concerned that it's getting over hyped. It's getting exactly the amount of "hype" it deserves. Docker has solved some major problems for application distribution. It removes OS allegiance from application support (meaning I don't have to install Ubuntu as my host OS to run an Ubuntu app), it provides a common framework for control and management, and it provides an open online repository of pre-built applications. Quote Link to comment
bonienl Posted April 28, 2015 Share Posted April 28, 2015 Dockers are the new kid on the block and they certainly have to prove themselves. True they have high potential but time will tell. To be clear - I do believe Dockers is the right way forward. A short story: I had a system locking issue a while back because the docker image run out of space (my own mistake in making it too small), it could be simply solved by enlarging the image size, but this story shows that dockers - though they are 'safer' - can have impact on the system as a whole. I have seen only one real issue with the System Stats plugin, that is when running over an extended period it can fill up the /var/log file system. Not because of System Stats ill behaving but simply because a 1 month rotation and associated logs doesn't fit in the standard system allocation of 128MB. The email "issue" only came to surface after introducing the notifications system, it is actually harmless (but annoying). It exists since the inception of System Stats 4 years ago, but always went unnoticed. It's my believe that System Stats is one of the most stable plugins out there. Quote Link to comment
jonp Posted April 28, 2015 Share Posted April 28, 2015 Dockers are the new kid on the block and they certainly have to prove themselves. True they have high potential but time will tell. To be clear - I do believe Dockers is the right way forward. A short story: I had a system locking issue a while back because the docker image run out of space (my own mistake in making it too small), it could be simply solved by enlarging the image size, but this story shows that dockers - though they are 'safer' - can have impact on the system as a whole. I'm mainly interested in issues that still persist today. Of course there were some bugs with Docker during the beta. It's bound to happen with a new technology, but for the most part, Docker has been far easier for us to implement and support than Plugins. I have never had someone call me (until this recent issue) with a container causing issues on the host. I have seen only one real issue with the System Stats plugin, that is when running over an extended period it can fill up the /var/log file system. Not because of System Stats ill behaving but simply because a 1 month rotation and associated logs doesn't fit in the standard system allocation of 128MB. The email "issue" only came to surface after introducing the notifications system, it is actually harmless (but annoying). It exists since the inception of System Stats 4 years ago, but always went unnoticed. Right, this was an issue and there were permissions issues with system stats too if you recall. These have been worked out, but at the time, there were folks having problems and didn't know the root cause. It's my believe that System Stats is one of the most stable plugins out there. This may be true today, but it wasn't this way all the time. Plugins get better with age, but only due to rigorous testing by the community. Quote Link to comment
bonienl Posted April 28, 2015 Share Posted April 28, 2015 This may be true today, but it wasn't this way all the time. Plugins get better with age, but only due to rigorous testing by the community. The same can be said for dockers, given time they will become rock solid. Like any newborn, it has to go through its paces ... Quote Link to comment
gundamguy Posted April 28, 2015 Share Posted April 28, 2015 This may be true today, but it wasn't this way all the time. Plugins get better with age, but only due to rigorous testing by the community. The same can be said for dockers, given time they will become rock solid. Like any newborn, it has to go through its paces ... Yes and lets not ignore the serious security issues that Docker had last fall. Quote Link to comment
limetech Posted April 28, 2015 Share Posted April 28, 2015 Here is the crux of the matter: at present unRaid OS primary market is "technology enthusiasts". There are exceptions, but for the most part, unRaid OS users, including most on this forum and certainly those chiming in on this topic, are very technically knowledgeable. In order for us to expand into other markets we need to make the product both easier to use and more secure. This is why we are taking a very hard look at how plugin development and security is handled. I apologize that the tone seems "heavy handed", that's not the intent. You all should know by now that we are very flexible and willing to change things and admit mistakes. Probably the right approach is to maintain a LimeTech sponsored package repo along with the idea of "tainted" plugins, something like what happens with linux kernel. At present however, we don't have this in place. Quote Link to comment
jonp Posted April 28, 2015 Share Posted April 28, 2015 This may be true today, but it wasn't this way all the time. Plugins get better with age, but only due to rigorous testing by the community. The same can be said for dockers, given time they will become rock solid. Like any newborn, it has to go through its paces ... Agreed. But I'd say that Docker has already quickly outpaced plugins. Just look at the sheer number of apps that just our community has created in Docker since it's inclusion in June of last year. Compare that to the # of plugins available and Docker crushes it. It's simply because Docker allows folks to use package managers inside containers to get the components they need to run an application. They are not dependent on us or someone else compiling dependencies for inclusion. This may be true today, but it wasn't this way all the time. Plugins get better with age, but only due to rigorous testing by the community. The same can be said for dockers, given time they will become rock solid. Like any newborn, it has to go through its paces ... Yes and lets not ignore the serious security issues that Docker had last fall. Meh, not really a concern. Every single piece of technology out there has security issues at some point or another. Bash and OpenSSL, cornerstones of the open source / cloud world, were proven to have major issues that make Docker's look like a scraped knee compared to a major head trauma. And how many of our users were effected by those issues? From my counting, it's zero. It's easy to google search "<insert tech here> security issue" and find something. Let's not get into a security debate about docker and plugins because that is an apples to oranges comparison. I still have yet to hear retorts to my bigger points of user control that we can provide with Docker vs. no control with plugins. Quote Link to comment
bonienl Posted April 28, 2015 Share Posted April 28, 2015 I think we are a bit derailing here, let's focus on the original question about (missing) packages and how to deal with that. I am open to suggestions Quote Link to comment
limetech Posted April 28, 2015 Share Posted April 28, 2015 I think we are a bit derailing here, let's focus on the original question about (missing) packages and how to deal with that. I am open to suggestions How many are there? We'll just include them in the build. You want sysstat? Ok. Probably we'll throw in openvpn as well. Not really opposed to python if we can prune that thing back a bit. That single package is almost as big as all the other code combined. We do still want to support servers with less memory. Quote Link to comment
jonp Posted April 28, 2015 Share Posted April 28, 2015 I think we are a bit derailing here, let's focus on the original question about (missing) packages and how to deal with that. I am open to suggestions How many are there? We'll just include them in the build. You want sysstat? Ok. Probably we'll throw in openvpn as well. Not really opposed to python if we can prune that thing back a bit. That single package is almost as big as all the other code combined. We do still want to support servers with less memory. I think the big two right now are Python and Perl as there are existing plugins that depend on them. Systat would be good as well. Quote Link to comment
bonienl Posted April 29, 2015 Share Posted April 29, 2015 I think we are a bit derailing here, let's focus on the original question about (missing) packages and how to deal with that. I am open to suggestions How many are there? We'll just include them in the build. You want sysstat? Ok. Probably we'll throw in openvpn as well. Not really opposed to python if we can prune that thing back a bit. That single package is almost as big as all the other code combined. We do still want to support servers with less memory. I have worked-out a proposal and asked Limetech to maintain a list of approved packages instead of working with an all-in-one master package. An update to the plugin manager is required for the new approach, this however will allow the plugin designers to choose only what is required without the extra burden. In the PLG file, it simply becomes a reference to the external packages which are required. The proposed format looks like this: <!-- External packages, list is maintained by Limetech. --> <FILE Name="/boot/packages" Run="upgradepkg --install-new"> <PTR>python</PTR> </FILE> <FILE Name="/boot/packages" Run="upgradepkg --install-new"> <PTR>perl</PTR> </FILE> The above example would download the appropriate packages "python" and "perl" to the folder "/boot/packages" and installs them from there. As a plugin designer you don't need to care about version numbering, this is maintained by the plugin manager (LT). This approach ensures that plugins will all use the same external packages, as indicated by LT. Quote Link to comment
SSD Posted April 29, 2015 Share Posted April 29, 2015 How about including bwm-ng unmenu and myMain make use of this to display real-time disk performance information, and seems appropriate to support web-guis. Quote Link to comment
bonienl Posted April 29, 2015 Share Posted April 29, 2015 How about including bwm-ng unmenu and myMain make use of this to display real-time disk performance information, and seems appropriate to support web-guis. Are these available as plugin (.PLG) ? Quote Link to comment
SSD Posted April 29, 2015 Share Posted April 29, 2015 Not sure. It is a very simple package download. I installed through unmenu's package manager, but a plugin might also have been developed. Quote Link to comment
Squid Posted April 29, 2015 Share Posted April 29, 2015 Not sure. It is a very simple package download. I installed through unmenu's package manager, but a plugin might also have been developed. Its in NerdPack Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.