Official unRAID Docker Image Discussion

[jonp]

The purpose of this post is to start a dialogue on the building of an official unRAID Docker image.  This image will serve as a template for container authors to use when building Dockerfiles for use on unRAID.

 

Overview

It is our intention with this image to maintain the contents therein on an ongoing basis to ensure that community developers have a stable and officially supported starting point for building their Docker containers.  We will not require the use of our base image in order to install a container into unRAID, but if a community developer elects to use an alternate "FROM" statement, they will be solely responsible for the ongoing use and maintenance of that variant, and the container will not be officially supported or endorsed by Lime Technology.  Again, this in no way means that we think alternative images are inferior, we simply need to limit our scope of direct supportability.

 

What Distro?

Quite simply, we are going with Ubuntu 14.04 as the distro for our image.  This isn't a point up for negotiation nor did this decision get made lightly.  Ubuntu represents the largest collection of supported containers today and many of the containers users leverage today are based on an Ubuntu base image already.  We also consulted with Docker on this directly and they too see Ubuntu as the largest player in their repositories right now.  Now we realize that some may not like this decision and will be tempted to reply here with a giant list of reasons why "X distro is better," but we're not going to entertain any discussions like that in this thread.  The purpose isn't to debate the distro we're choosing, but rather, what should be packaged within it from Lime Technology.

 

What about "best practices"?

From speaking with Docker directly, there are no formal "best practices" for our use case.  We are inventing a new use-case for Docker that hasn't been done before and as such, their recommendations to us have been to "do what works."  That means nothing is off limits from discussion.  We are open to all suggestions for package inclusions, but please explain why a specific package should be included.

 

What are we already planning to include?

The list of packages we already have planned for inclusion are as follows:

• Python 2.7
  
• Python 3
  
• Git (latest)
  
• Curl (latest)
  

Format for Replies

Please format your replies in such a way that let's us quickly identify what you're asking for and why.  Package name, version, and justification for inclusion are the main things to cover.  For justifications, citing other containers that already use this today and how is a really good start.

 

Maintaining the Image

Eric Schultz will be leading the charge on maintaining this image.  Eric is very good at keeping things up to date and will have a fairly regular release schedule.  The recommendation to container authors will be to always have their containers specify a version number of Eric's build.  This way version upgrades can be tested before being rolled out to the community at large.

 

If there are other questions about the base image, this is the place for them as well, but let's avoid any off-topic tirades about distro choice.  I know how quickly those can break out and that's not what we're looking to have happen here.

 

Thanks!

 

Update 10/14/2014

Here's an update on what's been discussed / is up for consideration:

 

Confirmed Additions

• nsenter
  

 

Up For Consideration

• Perl
  
• Docker enter
  

 

Update 10/16/2014

With the latest updates to Docker 1.3, the need for nsenter and docker-enter are pretty much eliminated, so we don't see ourselves adding these in our build.

[BRiT]

Simple question, how should docker creators manage the shortcomings of Ubuntu 14.04 for a docker environment?

 

Should they create their own base image off of that with all the needed fixes and then create their images from that? Is that how the phusion base image works already?

[jonp]

Simple question, how should docker creators manage the shortcomings of Ubuntu 14.04 for a docker environment?

 

Should they create their own base image off of that with all the needed fixes and then create their images from that? Is that how the phusion base image works already?

 

There are lots of goodies in Phusion, but there are others that we just don't agree with, such as SSH.  What we could do is make the Lime Tech image use a FROM statement of "phusion/baseimage:ver#" and then remove things from the image inside our Dockerfile (e.g. SSH).  That is an implementation detail that will get determined from this discussion.  Whether we use Phusion or specify Ubuntu directly in the FROM statement from within our Dockerfile, is a detail.  In the end, they are both Ubuntu.

 

I'd rather not get into too deep of a discussion about Phusion vs. Ubuntu in our "FROM" statement because that really shouldn't matter.  The bigger thing we want to gather here is what packages should be included.  In the end, we could determine that using Phusion is a good move or we could decide its not.  Bottom line is that we KNOW we want to have some things included in our image that are not included in Phusion by default.

 

Make sense?

[ljm42]

Hmm... What is the value in adding two versions of Python and Git to the base image, how is that better than just adding them to the containers that need them?  The three main dockers I use (Plex, plexWatch, Crashplan) will not benefit from those packages.

 

Depending on the answer to that question, Perl, PHP & Java may also be good candidates, and maybe a web server.  (I am not really arguing for this yet, but until I understand why anything should be in the base image, I can't really answer the question.)

 

--

 

In terms of moving away from phusion, plexWatch needs to run a cron job.  How will we handle this w/o phusion?

 

--

 

I agree that with nsenter there is zero need for SSH in a docker.  Would it make sense to add nsenter and docker-enter into unRAID by default?

[pinion]

The purpose of this post is to start a dialogue on the building of an official unRAID Docker image.  This image will serve as a template for container authors to use when building Dockerfiles for use on unRAID.

 

Overview

It is our intention with this image to maintain the contents therein on an ongoing basis to ensure that community developers have a stable and officially supported starting point for building their Docker containers.  We will not require the use of our base image in order to install a container into unRAID, but if a community developer elects to use an alternate "FROM" statement, they will be solely responsible for the ongoing use and maintenance of that variant, and the container will not be officially supported or endorsed by Lime Technology.  Again, this in no way means that we think alternative images are inferior, we simply need to limit our scope of direct supportability.

 

What's the point? What causes any other base image to not be stable? Why do I care if you officially support my base image? If I decide to make a docker for something that very few people are going to use, say pyTivo, and use this base image and something isn't working exactly what kind of support do you plan to provide?

 

This sounds overly critical of me and I'm sorry. Beta 6 has me so excited it's not even funny. I guess my bottom line question is this; What kind of support will you give if we use the unraid base image? All the other questions don't matter as long as anyone on any platform can use my docker image regardless. Also, props to Eric! I enjoy some of his containers.

 

 

 

[jonp]

 

 

 

What's the point? What causes any other base image to not be stable?  Why do I care if you officially support my base image? If I decide to make a docker for something that very few people are going to use, say pyTivo, and use this base image and something isn't working exactly what kind of support do you plan to provide?

 

First and foremost, if we had a common base image that LT maintains, we could include that image either in unRAID itself or make it auto download upon dockers first run on the system. Apps that point to that image will install faster as they won't need to re download that image.

 

In addition, do you get to provide input to phusion on the packages that they install in their image?  You will with LT.  If a bunch of authors ask us to include packages in the base image for you, we are going to try to accommodate that directly.

 

Bottom line:  more clear and direct line of communication with us at LT.

 

In addition, it is our goal to try and make popular and common containers available without having to specify a template repo.  The only containers that will be considered for this will be ones using our image template.

 

So in short, if you want to use a different image than ours, you can.  We are not going to force anyone to use this.

 

 

This sounds overly critical of me and I'm sorry. Beta 6 has me so excited it's not even funny.

 

Glad you were pumped for it!!

 

I guess my bottom line question is this; What kind of support will you give if we use the unraid base image?

 

1 - a frequently updated and maintained image with common and essential tools needed for the containers.

 

2 - possibility for your container to be officially endorsed by Lime Tech (increasing exposure)

 

3 - direct feedback with the image maintainers for ongoing communication and refinement.  We will listen to those that are making containers.

 

Basically we want to make it easier for the authors of Dockerfiles to get started with containers on unRAID and to be able to better endorse those community works. This is a logical step in that direction.

 

Thanks for the feedback and good direct questions!!

[jonp]

Hmm... What is the value in adding two versions of Python and Git to the base image, how is that better than just adding them to the containers that need them?  The three main dockers I use (Plex, plexWatch, Crashplan) will not benefit from those packages.

 

Some applications have a need for one version versus the other.  Others require git.  They are common components in various containers which is why we would want to include them.  These were Eric's "defaults" when I asked him what he wanted for sure in the image.

 

Depending on the answer to that question, Perl, PHP & Java may also be good candidates, and maybe a web server.  (I am not really arguing for this yet, but until I understand why anything should be in the base image, I can't really answer the question.)

 

The idea behind making an image in the first place is to accomplish a few goals as I outline in another post:

 

1)  Reduce the time it takes to install numerous applications.  If we all use a common image / version, it will make adding applications to the system overall faster.  Strength in numbers friends....

 

2)  Preemptively load the unRAID Docker image as a separate system task from adding your first container.  This will make the setup of Docker appear a little longer, but then adding apps from there would be much faster.  The first time experience for adding an app could cause a "judge a book by its cover" feeling and we'd like to have folks judge the true experience of speed of installing apps from the same base image over and over making each app take a lot less space overall but also less time to get to using it.

 

3)  Offer a route for official support and endorsement from Lime Tech.  An example of support could mean having more packages included in the Lime Tech image.  The more people using your Container and the more Containers like yours that add a package manually that we don't include by default, the more likely we will add it.  There is a symbiotic relationship between our motivations here.  We want to make this better for the user to minimize the amount of various snapshots in play on a system and we want to make it less burdensome for the community developer to have to manage the contents of their Dockerfile.  The more usage there is of a particular set of packages, the more likely we will include them into the image we provide.

 

4)  Offer an "upgrade when ready" approach to your own management of Containers as authors.  So let's say we release an image as version 1.0.  This image comes with a set of packages (package set A).  Now we see a bunch of people adding a new package over and over again for their own containers (application X).  If X occurs in a lot of downloads, we'll want to include that in package set A as a part of our image.  So we release 1.1 with updates to the packages for 1.0 and a new package for application X.  As a container author, i want my Dockerfile to state FROM limetech/unraid-base:1.0 until I've had time to adjust my image to use 1.1 and proven it works.  That's the beauty of Docker with revisioning like that.  All those revisions are stored on the disk and all will align to offer an experience with less IO, less capacity, and more performance at the same time.

 

5)  To try to help build a way for authors to collaborate on a common set of tools as opposed to everyone taking a different path. Provide some consistency and repeatability so we can start building some continuity into Container design.  I think there are some great ideas out there, but we need to hone in and focus on what's working, what's not, and ditch the rest.

 

In terms of moving away from phusion, plexWatch needs to run a cron job.  How will we handle this w/o phusion?

 

Will get back to you on this.  Need Eric's input.

 

I agree that with nsenter there is zero need for SSH in a docker.  Would it make sense to add nsenter and docker-enter into unRAID by default?

 

Sounds like a proposal for consideration if you ask me!

[NAS]

One other point to note on why we should do this ourselves. Phusion could go away. Our own base distro wont. Considering it is the rock all containers will sit I would prefer that we own it ourselves.

 

Thoughts and questions:

 

Will this be based on the existing Ubuntu official docker image. If so will we track point releases (currently 14.04.1, 14.04, trusty, latest (trusty/Dockerfile) or will be be building from scratch

 

What name and version numbering will be used i.e. limetech/ubuntu:14.04.1-XXX

 

I would like to see Perl included and if anyone requested specific modules we dont use CPAN but rather apt

 

I second the nsenter request. This is one area we could do something quite nice with perhaps some clever unRAID integration with screen maybe.. maybe even a web based console

 

One area that we need to work on is layer and updates. As I understand it every time you update (apt-get update) and then upgrade (apt-get upgrade) and then perhaps clean (apt-get clean ,remove, auto remove etc ) each action will generate a layer. These layer will be a dependency on later updates and could get quite messy and inefficient in terms of downloads. There are articles out there discussing rolling up layers vs. tags and I suggest we decide on this very early on.

 

 

[jonp]

One other point to note on why we should do this ourselves. Phusion could go away. Our own base distro wont. Considering it is the rock all containers will sit I would prefer that we own it ourselves.

 

Thoughts and questions:

 

Will this be based on the existing Ubuntu official docker image. If so will we track point releases (currently 14.04.1, 14.04, trusty, latest (trusty/Dockerfile) or will be be building from scratch

 

What name and version numbering will be used i.e. limetech/ubuntu:14.04.1-XXX

 

I would like to see Perl included and if anyone requested specific modules we dont use CPAN but rather apt

 

I second the nsenter request. This is one area we could do something quite nice with perhaps some clever unRAID integration with screen maybe.. maybe even a web based console

 

One area that we need to work on is layer and updates. As I understand it every time you update (apt-get update) and then upgrade (apt-get upgrade) and then perhaps clean (apt-get clean ,remove, auto remove etc ) each action will generate a layer. These layer will be a dependency on later updates and could get quite messy and inefficient in terms of downloads. There are articles out there discussing rolling up layers vs. tags and I suggest we decide on this very early on.

 

Couple of things:

 

1)  Eric agrees on nsenter completely, that's getting added.

2)  As for limetech repo/version naming convention, it will be something like limetech/baseimage:20141014 (we will use the date without dashes or dots for version).  Eric did mention that the most frequent you'd see us updating this would probably be weekly, but there may be times where it's a little less frequent (e.g. when there are no new updates).

3)  As for point releases, we are going to base our image off another image (either the official Ubuntu base image or something like Phusion) and then modify it from there.  We will keep up with point releases from Ubuntu as they become available and merged into whatever components we use from the upstream.  That said, we will NEVER specify the version as "latest" in our base image.  Our image will always contain a specific hash for a version of the upstream.  We recommend all authors do the same.  Using "latest" is just not a good practice.

4)  As for Perl, we're thinking about that, but need to check how big it will impact the image and how many containers will use it.

5)  As far as layers and updates, as a best practice, we would like to combine as many apt commands merged into as few of run statements as is possible.  E.g. collapsing apt-get update and apt-get install onto the same "RUN" statement.  Each "RUN" statement equates to a layer, so reducing this, would greatly diminish the problems with layered complexity over time.  That said, this is only a solution to addressing layer sprawl from the container author's perspective.  From our side it's a little more challenging.  Eric and I have talked about the possibility of breaking out updates from upgrades where an update would be a change of the image from lime tech that includes either a distro-based point release or a specific package update.  Then upgrades would be when there is a new major release from the base (such as a new base version of ubuntu, not just a point release).  If we took this approach, upgrades would be a little more "intensive", but would ideally flatten out a lot of the sprawl of layers.  Updates would be less intensive and could be applied more easily "in place".  We're still thinking through this and are open to suggestions from the container authors out there, but this is going to take some teamwork and cooperation to properly define.

[NAS]

Sounds sensible and like a WIP.

 

Dont disagree with anything other than the naming convention "limetech/baseimage:20141014" since it would require you to get on the internet to find out what specific Ubuntu version each one is. Also it hides the fact that a single number change could be a single package update a full OS point release (14.04 to 14.04.1) or even a major release (12.04 to 14.04).

 

I would suggest that you reflect the donor OS version in the numbering even if it is long winded like "limetech/baseimage:14.04.1-20141014"

[jonp]

Didn't think about that for the naming convention.  Will run that by Eric.

 

And this is definitely a WIP.  We have a pretty good idea on how we should handle this, but want more feedback from container authors to help shape it further.

[NAS]

I was thinking that probably the most important question right now we need to ask is what does Phusion have that we need. Once we have that we can add to our image and we can start porting.

[pinion]

When can we expect the initial release of this image? I'm holding off on doing a Madsonic container until this is out.

[jonp]

When can we expect the initial release of this image? I'm holding off on doing a Madsonic container until this is out.

Need more feedback from the other devs here before I can speak to a release time. We want to get cracking on this asap.

[NAS]

When can we expect the initial release of this image? I'm holding off on doing a Madsonic container until this is out.

 

Can I suggest you develop using ubuntu:14.04 and feed back as you identify components you require. The Limetech can push out a new base OS and modify it on the fly based on your requirements.

[gfjardim]

OK, I'm arriving late to the party, but have some ideas to offer:

 

1) we have to add a process management system - probably Supervisor is the way to do it;

2) I would start with Debian and not Ubuntu because of image size;

3) whatever base image is used, we must enforce it with the correct umask, nobody user and users group set to correctly ID;

 

[BRiT]

3) whatever base image is used, we must enforce it with the correct umask, nobody user and users group set to correctly ID;

 

Along with that, I would like to add that it should respect the settings of the unraid hosts /etc/resolv.conf as well. Along with other typical cleanups is setting it so the package manager is not in interactive mode.

 

As for Debian instead of Ubuntu, how's the package management for debian? It seems like Ubuntu apt-get repositories are very well maintained and populated, why switch away from that benefit just for a very minor one time only space savings? Don't throw the baby out with the bath water.

[gfjardim]

3) whatever base image is used, we must enforce it with the correct umask, nobody user and users group set to correctly ID;

 

Along with that, I would like to add that it should respect the settings of the unraid hosts /etc/resolv.conf as well. Along with other typical cleanups is setting it so the package manager is not in interactive mode.

 

As for Debian instead of Ubuntu, how's the package management for debian? It seems like Ubuntu apt-get repositories are very well maintained and populated, why switch away from that benefit just for a very minor one time only space savings? Don't throw the baby out with the bath water.

 

From the repository perspective, it's practically the same of Debian's based Ubuntu. The size difference is not negligible; my ownCloud container uses 788MB using Phusion, and 415MB using Debian Wheezy.

 

 

[BRiT]

3) whatever base image is used, we must enforce it with the correct umask, nobody user and users group set to correctly ID;

 

Along with that, I would like to add that it should respect the settings of the unraid hosts /etc/resolv.conf as well. Along with other typical cleanups is setting it so the package manager is not in interactive mode.

 

As for Debian instead of Ubuntu, how's the package management for debian? It seems like Ubuntu apt-get repositories are very well maintained and populated, why switch away from that benefit just for a very minor one time only space savings? Don't throw the baby out with the bath water.

 

From the repository perspective, it's practically the same of Debian's based Ubuntu. The size difference is not negligible; my ownCloud container uses 788MB using Phusion, and 415MB using Debian Wheezy.

 

If you're really concerned about space, then none of the Docker images should have the capabilities of auto-updating. The "make" and "gcc" packages are well over 100 Megs! All other developer tools should be stripped out as well, ie: anything that lets you build or compile.

 

Also, all the steps of your Docker file should be done in a single step as well to minimize the layer commits. These steps are best contained in a separate script, lets call it "build.sh". It is this build.sh script that has all the fun of apt-get, wget, curl, make, etc.

 

# Add build script

ADD build.sh /opt/

RUN chmod +x /opt/build.sh

 

# invoke build script

RUN /opt/build.sh

 

Have you compared size differences by including the following steps at the very end of your Docker build file? This is stripping out all the items that are only needed once at docker build time. This sort of thing has shaved off 150 Megs on my own personal Dockers.

 

# remove compile-time only items

apt-get remove -y tcl-dev libssl-dev make gcc

apt-get autoremove -y

apt-get autoclean -y

 

[sparklyballs]

Can I suggest these two for inclusion in your image.

 

software-properties-common python-software-properties

 

to give the ability to easily add in any ppa's that may be required for certain apps and deps of those apps.