Removing Python and Perl from unRAID 6 bzroot

[gfjardim]

EDIT:  JONP HERE!  I AM MODIFYING GF's POST AS I SPLIT THIS TOPIC FROM ANOTHER THREAD

 

With our plan to remove Xen from unRAID 6 in an upcoming release, this also means the removal of two large packages from our bzroot:  Python and Perl.  I realize that some plugins make use of these dependencies today.  The purpose of this thread is to discuss the impact this will have and what solutions exist beyond just adding them back to bzroot (which isn't going to happen unless core OS functionality depends on it).

[jonp]

Python and Perl are extremely relevant, so even if you remove Xen from unRAID, these must stay.

They are gone. We have a build without them. They can be readded via the boot/extra folder.

 

Unraid OS is not dependent on these. Just plugins.

[gfjardim]

Python and Perl are extremely relevant, so even if you remove Xen from unRAID, these must stay.

They are gone. We have a build without them. They can be readded via the boot/extra folder.

 

Unraid OS is not dependent on these. Just plugins.

This would make my live worse. 

 

Please advertise this when release the update.

[jonp]

Python and Perl are extremely relevant, so even if you remove Xen from unRAID, these must stay.

They are gone. We have a build without them. They can be readded via the boot/extra folder.

 

Unraid OS is not dependent on these. Just plugins.

This would make my live worse. 

 

Please advertise this when release the update.

How so?  They can be readded outside of the bzroot. It removed two very large dependencies from our build. 

[gfjardim]

Python and Perl are extremely relevant, so even if you remove Xen from unRAID, these must stay.

They are gone. We have a build without them. They can be readded via the boot/extra folder.

 

Unraid OS is not dependent on these. Just plugins.

This would make my live worse. 

 

Please advertise this when release the update.

How so?  They can be readded outside of the bzroot. It removed two very large dependencies from our build.

 

That's why. I didn't mean terrible worse, just worse. 

[trurl]

Python and Perl are extremely relevant, so even if you remove Xen from unRAID, these must stay.

They are gone. We have a build without them. They can be readded via the boot/extra folder.

 

Unraid OS is not dependent on these. Just plugins.

This would make my live worse. 

 

Please advertise this when release the update.

How so?  They can be readded outside of the bzroot. It removed two very large dependencies from our build.

 

That's why. I didn't mean terrible worse, just worse. 

Wasn't python (or maybe it was perl) one of the dependency conflicts that happened with some of the plugins on v5? I seem to remember seeing syslogs where some plugin would install one version, then another plugin would replace it with an older version.

[jonp]

Python and Perl are extremely relevant, so even if you remove Xen from unRAID, these must stay.

They are gone. We have a build without them. They can be readded via the boot/extra folder.

 

Unraid OS is not dependent on these. Just plugins.

This would make my live worse. 

 

Please advertise this when release the update.

How so?  They can be readded outside of the bzroot. It removed two very large dependencies from our build.

 

That's why. I didn't mean terrible worse, just worse. 

Wasn't python (or maybe it was perl) one of the dependency conflicts that happened with some of the plugins on v5? I seem to remember seeing syslogs where some plugin would install one version, then another plugin would replace it with an older version.

 

Python is used by a number of plugins including Couchpotato, Sickbeard, etc.  Those types of apps could be more easily added / maintained as Docker Containers. For those that still want the plugin version, for whatever reason, they will need to add Python to their systems.  This could be done by a plugin author directly or a new plugin could be created to do this.  Supporting two methods of installing the same app doesn't make sense to us, so we're not going to keep Python in unRAID just for the sake of supporting plugins which have Docker Container alternatives available.

[bonienl]

Python and Perl are extremely relevant, so even if you remove Xen from unRAID, these must stay.

They are gone. We have a build without them. They can be readded via the boot/extra folder.

 

Unraid OS is not dependent on these. Just plugins.

This would make my live worse. 

 

Please advertise this when release the update.

How so?  They can be readded outside of the bzroot. It removed two very large dependencies from our build.

 

It's true they can be added as a separate package, but it is a PITA.

 

For example the Dynamix temperature plugin relies on the existence of perl to do driver scanning. The plugin can be revised to an installation of perl when missing, but this is going back to the old v5 days where different plugins would install different versions and cause all kind of conflicts.

 

The 'beauty' of these packages already installed is that potential conflicts can be avoided altogether. I am wondering if the "space" saving weigh up to the inconveniences I foresee ?

 

[jonp]

 

 

Python and Perl are extremely relevant, so even if you remove Xen from unRAID, these must stay.

They are gone. We have a build without them. They can be readded via the boot/extra folder.

 

Unraid OS is not dependent on these. Just plugins.

This would make my live worse. 

 

Please advertise this when release the update.

How so?  They can be readded outside of the bzroot. It removed two very large dependencies from our build.

 

It's true they can be added as a separate package, but it is a PITA.

 

For example the Dynamix temperature plugin relies on the existence of perl to do driver scanning. The plugin can be revised to an installation of perl when missing, but this is going back to the old v5 days where different plugins would install different versions and cause all kind of conflicts.

 

The 'beauty' of these packages already installed is that potential conflicts can be avoided altogether. I am wondering if the "space" saving weigh up to the inconveniences I foresee ?

 

What plugins exist that aren't available as docker that require python or Perl?  You've got one listed so far, curious how many others. Maybe a master plugin with python and Perl could be referenced by other plugins that share the dependency requirement.

[bonienl]

 

 

Python and Perl are extremely relevant, so even if you remove Xen from unRAID, these must stay.

They are gone. We have a build without them. They can be readded via the boot/extra folder.

 

Unraid OS is not dependent on these. Just plugins.

This would make my live worse. 

 

Please advertise this when release the update.

How so?  They can be readded outside of the bzroot. It removed two very large dependencies from our build.

 

It's true they can be added as a separate package, but it is a PITA.

 

For example the Dynamix temperature plugin relies on the existence of perl to do driver scanning. The plugin can be revised to an installation of perl when missing, but this is going back to the old v5 days where different plugins would install different versions and cause all kind of conflicts.

 

The 'beauty' of these packages already installed is that potential conflicts can be avoided altogether. I am wondering if the "space" saving weigh up to the inconveniences I foresee ?

 

What plugins exist that aren't available as docker that require python or Perl?  You've got one listed so far, curious how many others. Maybe a master plugin with python and Perl could be referenced by other plugins that share the dependency requirement.

 

The question is not about absolute numbers of plugins using one or the other, it is more about how to deal with the plugins if they do need either python or perl.

 

The nice thing about v6 was (is?) that many dependencies have been taken out by including the relevant packages in the distro. If that is changed a good alternative must be available. Do you think a master plugin is the right answer ?

 

[jonp]

The question is not about absolute numbers of plugins using one or the other, it is more about how to deal with the plugins if they do need either python or perl.

 

It's not the numbers, it's which ones.  I would wager most plugins that require Python or Perl are available in container form.  I'm interested in those that are not available as a container but have the dependency requirements.  I would also wager that most functionality could be obtained through other methods than Python or Perl.

 

The nice thing about v6 was (is?) that many dependencies have been taken out by including the relevant packages in the distro. If that is changed a good alternative must be available. Do you think a master plugin is the right answer ?

 

A master plugin is one possible solution and it's a much cleaner one, but it's not the only method.  I'm open to hearing alternatives.

 

I realize that from a 3rd party developer standpoint, removing these seems restrictive or limiting, but that's just one perspective.  From our standpoint, every component we add to the OS is something we are responsible to maintain and support.  Bug fixes, security patches, and configuration issues alike.  Sometimes when packages are upgraded, significant underpinnings are changed which don't compile nicely for us (the latest xfsprogs are a prime example of that).  This can cause delays in putting out new releases.  The less we have to maintain, the more we can focus on feature additions and more frequent releases.

 

I'm going to fork this discussion about Perl and Python to the dev board as it's a separate topic of discussion than Xen.

[bonienl]

I like to put this in a bit wider discussion, inevitably at some point some package is required (though admittedly with Dockers this is less likely going to happen).

 

We have learned from v5 that it isn't a good idea to let plugins install themselves other packages, different package versions and conflicts are the result. Thinking out loud - one approach maybe a central repository which can be used by plugins to get/install the required package(s), as opposed to a master plugin getting the packages instead. Or a combination of both - a master plugin retrieving packages from a central repository.

 

The crux here is - who is going to maintain all of this and make sure that things stay up to date and complete ?

 

Something outdated or incomplete, soon is doomed to be ignored and people follow their own road, that will be history repeating itself as seen with v5.

 

Maybe I am a bit too skeptical here and the situation isn't that bad, at this point in time Dockers still need to prove they are better/easer/faster to setup and use than traditional plugins, which I hope will happen, but until then people will continue using plugins (these are still actively developped) and only switch when benefits are clear to everyone.

 

[jonp]

I like to put this in a bit wider discussion, inevitably at some point some package is required (though admittedly with Dockers this is less likely going to happen).

 

We have learned from v5 that it isn't a good idea to let plugins install themselves other packages, different package versions and conflicts are the result. Thinking out loud - one approach maybe a central repository which can be used by plugins to get/install the required package(s), as opposed to a master plugin getting the packages instead. Or a combination of both - a master plugin retrieving packages from a central repository.

 

The crux here is - who is going to maintain all of this and make sure that things stay up to date and complete ?

 

Something outdated or incomplete, soon is doomed to be ignored and people follow their own road, that will be history repeating itself as seen with v5.

 

Maybe I am a bit too skeptical here and the situation isn't that bad, at this point in time Dockers still need to prove they are better/easer/faster to setup and use than traditional plugins, which I hope will happen, but until then people will continue using plugins (these are still actively developped) and only switch when benefits are clear to everyone.

 

I think the situation is far better than you think.  Look at the sheer number of available containers at this point that our community has created in < 1 year compared to how many plugins exist.  Yes, there are probably some containers that aren't up to snuff totally just yet, but most definitely are.  And for those that don't want containers, VMs are available and still a better solution than Plugins.

 

I realize that Plugins are what brought us Dynamix to begin with, but your use case is a rare one compared to the majority of plugins that are essentially just installing 3rd party apps like Plex, Dropbox, etc.

 

We already have enough packages to maintain on our own with the OS itself, we don't need to be maintaining even more that have nothing to do with unRAID OS.  If someone wants to carry that torch, great, we won't stop them, but it won't be Lime Technology.  The only packages we want to support are those required by the OS to support the functionality we deliver.

[HellDiverUK]

UnRAID is using a paltry 110MB of my 8GB USB stick.  Removing python and perl isn't really a deal breaker as far as size goes is it?  The 8GB stick is new, and it was literally the smallest brand name stick I could get.

[trurl]

UnRAID is using a paltry 110MB of my 8GB USB stick.  Removing python and perl isn't really a deal breaker as far as size goes is it?  The 8GB stick is new, and it was literally the smallest brand name stick I could get.

I think the argument is more about memory footprint, and probably more importantly, package support.

[BRiT]

I would be fine with removing perl and python from the main unraid os if and only if the plugin system has dependency and version support built in.

 

Without both of those, this makes for a metric mess for plugin developers.

 

Also, didnt some of the newly created and useful checksum programs require one or the other? @weebotech

[HellDiverUK]

UnRAID is using a paltry 110MB of my 8GB USB stick.  Removing python and perl isn't really a deal breaker as far as size goes is it?  The 8GB stick is new, and it was literally the smallest brand name stick I could get.

I think the argument is more about memory footprint, and probably more importantly, package support.

 

Again, 4GB RAM is about as low as you can realistically go with anything that isn't ancient.

[overbyrn]

I like to put this in a bit wider discussion, inevitably at some point some package is required (though admittedly with Dockers this is less likely going to happen).

 

We have learned from v5 that it isn't a good idea to let plugins install themselves other packages, different package versions and conflicts are the result. Thinking out loud - one approach maybe a central repository which can be used by plugins to get/install the required package(s), as opposed to a master plugin getting the packages instead. Or a combination of both - a master plugin retrieving packages from a central repository.

 

The crux here is - who is going to maintain all of this and make sure that things stay up to date and complete ?

 

Something outdated or incomplete, soon is doomed to be ignored and people follow their own road, that will be history repeating itself as seen with v5.

 

Maybe I am a bit too skeptical here and the situation isn't that bad, at this point in time Dockers still need to prove they are better/easer/faster to setup and use than traditional plugins, which I hope will happen, but until then people will continue using plugins (these are still actively developped) and only switch when benefits are clear to everyone.

 

I think the situation is far better than you think.  Look at the sheer number of available containers at this point that our community has created in < 1 year compared to how many plugins exist.  Yes, there are probably some containers that aren't up to snuff totally just yet, but most definitely are.  And for those that don't want containers, VMs are available and still a better solution than Plugins.

 

I realize that Plugins are what brought us Dynamix to begin with, but your use case is a rare one compared to the majority of plugins that are essentially just installing 3rd party apps like Plex, Dropbox, etc.

 

We already have enough packages to maintain on our own with the OS itself, we don't need to be maintaining even more that have nothing to do with unRAID OS.  If someone wants to carry that torch, great, we won't stop them, but it won't be Lime Technology.  The only packages we want to support are those required by the OS to support the functionality we deliver.

 

A step in the wrong direction in my opinion. 

 

JonP, it's very clear from this thread that you're a big proponent of Docker containers over plugins. Fair enough, that's your prerogative. Others may not wish to embrace that aspect yet and perhaps would rather stick to plugins. If you're going to do a 180 on what Limetech currently bakes into the build, then perhaps you go the whole hog and remove the plugin architecture altogether? Too strong? Probably, but it feels as though Limetech is doing an about face from what it previously recognized as an in issue in v5.

 

I agree with Bonienl, the problem with maintaining Python and Perl outside core unRAID is the possibilities of one plugin clashing with another. If we ignore the 'use containers instead' argument for a moment, I'd like to give an example of one such case...

 

At the point which most plugins standardized on Python 2.7, there came an issue where a number of plugins were using a build of Python widely distributed by forum member piotrasd. One the one hand this worked well as the build contained a few extras over a native python package from more traditional channels.  But the other side of the coin was that it caused headaches for other plugin authors who chose to use python packages from more regular distribution channels. Do a search for python-2.7.3-i686-5PTr.txz. You'll soon see what I mean. The piotrasd version contained python-cheetah, which would usually be a separate dependency. Easy enough to fix, but a pain all the same.

 

I think expecting someone in the community to maintain a master plugin is the wrong way to go about it. I understand you do not wish to bloat the unRAID image and only want to bundle what is needed for unRAID to function, but a balance has to be found and currently it feels as this action is as much about engineering folks onto containers as it is anything else. 

 

[jonp]

I like to put this in a bit wider discussion, inevitably at some point some package is required (though admittedly with Dockers this is less likely going to happen).

 

We have learned from v5 that it isn't a good idea to let plugins install themselves other packages, different package versions and conflicts are the result. Thinking out loud - one approach maybe a central repository which can be used by plugins to get/install the required package(s), as opposed to a master plugin getting the packages instead. Or a combination of both - a master plugin retrieving packages from a central repository.

 

The crux here is - who is going to maintain all of this and make sure that things stay up to date and complete ?

 

Something outdated or incomplete, soon is doomed to be ignored and people follow their own road, that will be history repeating itself as seen with v5.

 

Maybe I am a bit too skeptical here and the situation isn't that bad, at this point in time Dockers still need to prove they are better/easer/faster to setup and use than traditional plugins, which I hope will happen, but until then people will continue using plugins (these are still actively developped) and only switch when benefits are clear to everyone.

 

I think the situation is far better than you think.  Look at the sheer number of available containers at this point that our community has created in < 1 year compared to how many plugins exist.  Yes, there are probably some containers that aren't up to snuff totally just yet, but most definitely are.  And for those that don't want containers, VMs are available and still a better solution than Plugins.

 

I realize that Plugins are what brought us Dynamix to begin with, but your use case is a rare one compared to the majority of plugins that are essentially just installing 3rd party apps like Plex, Dropbox, etc.

 

We already have enough packages to maintain on our own with the OS itself, we don't need to be maintaining even more that have nothing to do with unRAID OS.  If someone wants to carry that torch, great, we won't stop them, but it won't be Lime Technology.  The only packages we want to support are those required by the OS to support the functionality we deliver.

 

A step in the wrong direction in my opinion. 

 

JonP, it's very clear from this thread that you're a big proponent of Docker containers over plugins. Fair enough, that's your prerogative. Others may not wish to embrace that aspect yet and perhaps would rather stick to plugins. If you're going to do a 180 on what Limetech currently bakes into the build, then perhaps you go the whole hog and remove the plugin architecture altogether? Too strong? Probably, but it feels as though Limetech is doing an about face from what it previously recognized as an in issue in v5.

 

I agree with Bonienl, the problem with maintaining Python and Perl outside core unRAID is the possibilities of one plugin clashing with another. If we ignore the 'use containers instead' argument for a moment, I'd like to give an example of one such case...

 

At the point which most plugins standardized on Python 2.7, there came an issue where a number of plugins were using a build of Python widely distributed by forum member piotrasd. One the one hand this worked well as the build contained a few extras over a native python package from more traditional channels.  But the other side of the coin was that it caused headaches for other plugin authors who chose to use python packages from more regular distribution channels. Do a search for python-2.7.3-i686-5PTr.txz. You'll soon see what I mean. The piotrasd version contained python-cheetah, which would usually be a separate dependency. Easy enough to fix, but a pain all the same.

 

I think expecting someone in the community to maintain a master plugin is the wrong way to go about it. I understand you do not wish to bloat the unRAID image and only want to bundle what is needed for unRAID to function, but a balance has to be found and currently it feels as this action is as much about engineering folks onto containers as it is anything else.

 

overbyrn,

 

Thank you for your input.  Between you, gfjardim, bonienl, and a few others, it's clear that this is going to make things more difficult and we don't want to do that, so we did discuss this internally and agreed that it's in everyone's best interests for us to continue to support python and perl...in one way or another.  One option we have discussed is to create a plugin (official LT plugin) that does just that.  More on this next week.  Gotta go run some errands now...

 

- jon

[RobJ]

I would just like to say I really appreciate the tone of most of these discussions.  They're frank and honest yet respectful, all sides represented, and clearly show all parties are looking for what's best for the future, for the overall good of the project.  I consider this very healthy.

[limetech]

Here are the RAM footprints of three builds:

 

1. NAS functionality and Docker (no KVM and no Xen):

 

251MB

 

2. NAS functionality, Docker, and KVM (no Xen):

 

285MB  (size increase due to inclusion of libvirt and qemu)

 

3. NAS functionality, Docker, KVM, and Xen:

 

430MB (increase due to python and perl)

 

Each of these builds can be it's own download zip file.

 

The only reason we added python and perl was because they are needed for Xen management.  We intend to deprecate Xen support starting with next release, and then hopefully drop it entirely in the next release after that.

 

Re: Plugins - Our official policy is that we highly discourage their use.  There are two main reasons for this:

1. Plugins are inherently a huge security risk.  They are installed with 'root' privilege.  Many plugins install 3rd party packages from who-knows-where containing who-knows-what.

2. Plugins will always suffer from problems with incompatible dependencies.  unRaid OS is not a distro like say Arch or debian.

 

We have spent a great deal of effort refining replacements to the "plugin" system: docker containers and virtual machines.  Using these technologies our goal is to reduce the core OS even further.

 

I can see 'plugins' still being useful for webGui enhancement (where all the source is visible), but not for adding executables.

[JonathanM]

Re: Plugins - Our official policy is that we highly discourage their use.  There are two main reasons for this:

1. Plugins are inherently a huge security risk.  They are installed with 'root' privilege.  Many plugins install 3rd party packages from who-knows-where containing who-knows-what.

Is this statement evidence of a change in the basic philosophy of unraid? Is security starting to get some attention?

 

Re: http://lime-technology.com/forum/index.php?topic=39098.msg365932#msg365932

 

Whenever basic security has been brought up in the past, the implied message was that security took a back seat to ease of use and integration into the home LAN.

[jonp]

Re: Plugins - Our official policy is that we highly discourage their use.  There are two main reasons for this:

1. Plugins are inherently a huge security risk.  They are installed with 'root' privilege.  Many plugins install 3rd party packages from who-knows-where containing who-knows-what.

Is this statement evidence of a change in the basic philosophy of unraid? Is security starting to get some attention?

 

Re: http://lime-technology.com/forum/index.php?topic=39098.msg365932#msg365932

 

Whenever basic security has been brought up in the past, the implied message was that security took a back seat to ease of use and integration into the home LAN.

In v6 we are taking a more active approach to security patching as more and more users add Internet facing services to their servers. Throughout the beta, we have been more relaxed on this because we have been focused on feature enhancements and bug fixes for core functionality.  That's one of the reasons it has been a beta. As we wrap beta up in prep for rc/final, you can expect a more frequent patching schedule for security.

[jonp]

Also, there is a big difference between the security risk imposed by what outsiders can do without an invitation and what a plugin could do that is invited by user invocation.

[bonienl]

Re: Plugins - Our official policy is that we highly discourage their use.  There are two main reasons for this:

1. Plugins are inherently a huge security risk.  They are installed with 'root' privilege.  Many plugins install 3rd party packages from who-knows-where containing who-knows-what.

2. Plugins will always suffer from problems with incompatible dependencies.  unRaid OS is not a distro like say Arch or debian.

 

One way to mitigate risks with plugins is to work with certified plugins only. I know this will give an addiional burden but at the end of the day it will give ensurance to both LT and the end-user that a particular plugin is suitable and compatible.

 

Certified plugins should be listed (downloadable) from a managed site, this will make it easy to find them and may give the oppertunity to distinguish between plugins and dockers, read: give preferential advice.

 

I understand that all focus is now on Docker and VM, but I believe there is still place and reason to have plugins. Not all plugins require third party software. The statement highly discourage doesn't sound like an invitation to continue on plugins, while they are still very useful. Knowing how much time and effort I've put in developping plugins (and the plugin manager), your announcement is 'bitter sweet'.