[POLL] Docker Container Organisation for the unRAID community



Recommended Posts

I had posted this before, but that thread has since been deleted. I've been using this as the base image for a lot of apps and it has worked out very well so far: https://github.com/phusion/baseimage-docker

 

Care to share your dockers?  I've looked at creating some dockers with that as a base, but just can't find the time to finish.

 

I'm using SAB & NzbDrone now, and will probably look into adding a torrent app one of these days.

 

I hate to add to the noise, especially in a thread about moderator approval for dockers ;) But it's quite simple, their docs give a template Dockerfile to use with a note on where to add your own build instructions. So all I've done so far is to take any Ubuntu docker, change the base image to phusion's and then put the body of the ubuntu file into the appropriate section.

 

Perhaps later I could post them in another thread. But I'm also still playing around with some concepts/config and they just released a new tagged version today I need to checkout.

Link to comment
  • Replies 51
  • Created
  • Last Reply

Top Posters In This Topic

Top Posters In This Topic

 

I wasn't aware of that Docker Image. Yes, that guy has gone through the trouble to clean up all the docker related issues that Debian based Distros (Ubuntu / Mint / etc.) have with Docker.

 

As far as the package manager is concern... I would like people to start naming apps that Ubuntu has that other Distros don't.

 

If you say Arch doesn't have Plex Media Server... it does. Yes, it is in AUR (which is kind of like adding a Repo to Ubuntu but way better) but even a noob can install it very easily.

 

To install PMS in Arch Linux:

 

yaourt -S plexmediaserver

 

It's worth noting that phusion isn't some random guy, it's the company behind Passenger among other things. As far as distros go, it's great you want to run with the one's you've mentioned. I'm going in a different direction though.

Link to comment

who is interested in doing a few proof of concept docks in TCL and accept the burden of helping users out with it for at least a short period?

 

Tell me a few apps you want and I will create them. Takes all of 5 - 10 minutes to create a TCL one.

 

Debian based distros SUCK for Docker due to sysinitv (upstart) and even more so with Apps like CouchPotato, Sickbeard, etc. that update themselves through GIT. It will become clear when people get sick of not being notified when their docker apps stopped and it is unable to restart by itself. Example: Try updating CouchPotato, Sickbeard, etc. You either have to restart the container or rebooting unRAID.

 

I swear I think a lot of the unRAID users have "Stockholm Syndrome" due to all the beatings by Slackware. If Linux isn't complicated and harder than it needs to be (unRAID / Slackware)... People don't think there machine loves them or reliable.

 

Now that is a valid reason for moving from Debian to something else. We can argue over saving 100mb of cache disk space, but not over something like a web application failing as it asks to be updated...

Link to comment

It's worth noting that phusion isn't some random guy, it's the company behind Passenger among other things. As far as distros go, it's great you want to run with the one's you've mentioned. I'm going in a different direction though.

 

I'm not really even talking to you. I am trying to educate the other users who are creating Docker images in Debian based Distros that are not dockerized. You are using a good base image for your Docker OS. Sure, it has some bloat but with the size of drives / speed of our internet connections does this really matter? I think for most people it doesn't.

 

As far as the tweaks that need to be done to do Debian based Distros (Ubuntu, Mint, etc) in Docker if you care about your containers:

 

The init process -- PID 1 -- inherits all orphaned child processes and must reap them. Most Docker containers do not have an init process that does this correctly. As a result their containers become filled with zombie processes over time.

 

Furthermore, docker stop sends SIGTERM to the init process, which is then supposed to stop all services. Unfortunately most init systems don't do this correctly within Docker since they're built for hardware shutdowns instead. This causes processes to be hard killed with SIGKILL, which doesn't give them a chance to correctly deinitialize things. This can cause file corruption.

 

APT incompatibilities with Docker (See https://github.com/dotcloud/docker/issues/1024)

 

syslog-ng - A syslog daemon is necessary so that many services - including the kernel itself - can correctly log to /var/log/syslog. If no syslog daemon is running, a lot of important messages are silently swallowed. Only listens locally.

 

logrotate - Rotates and compresses logs on a regular basis.

 

ssh server - Allows you to easily login to your container to inspect or administer things.

 

cron - The cron daemon must be running for cron jobs to work.

 

Replace Upstart - Either runit or systemd. Upstart has all kinds of problems with docker and it doesn't have a service supervision or management. With runit or systemd you can restart daemons when they crash or they need to restart after an upgrade (couchpotato, sickbeard, etc). Otherwise you have to manually restart the docker container or rebooting unRAID.

Link to comment

I had posted this before, but that thread has since been deleted. I've been using this as the base image for a lot of apps and it has worked out very well so far: https://github.com/phusion/baseimage-docker

 

I like that it's based on ubuntu, so the app repositories are available. It also avoids upstart which, as grumpy mentioned, has a lot of issues with docker. They've done quite a bit to make it compatible with docker, and it is currently the #3 most popular repository on docker's hub.

 

I'm all for thin versions of Linux, but at the same time, the bigger distros tend to have more packages available and there's less fussing about to get some random app working unless you want to go down the compile path.

 

Not really trying to convince anyone or fuel a distro war. I'm sold on it for now and thought others might like the approach.

 

I wasn't aware of that Docker Image. Yes, that guy has gone through the trouble to clean up all the docker related issues that Debian based Distros (Ubuntu / Mint / etc.) have with Docker.

 

As far as the package manager is concern... I would like people to start naming apps that Ubuntu has that other Distros don't.

 

If you say Arch doesn't have Plex Media Server... it does. Yes, it is in AUR (which is kind of like adding a Repo to Ubuntu but way better) but even a noob can install it very easily.

 

To install PMS in Arch Linux:

 

yaourt -S plexmediaserver

 

OK alibi / reminder: I've never said Arch didn't have Plex.  I said it isn't in the normal repo's and we've been dependent on someone else to build it from AUR.  A state of affairs reminiscent of plug-in's dependance on the dev to keep things up to date (all be it a harder job than in Arch).

 

So I'm about to try yaourt (I'm assuming I need to install yaourt too?).  I know Ironic said something about in another thread but a quick look at yaourt didn't fill me with confidence and I didn't want to bug ironic around things I should be able to learn myself.  But if it really is that easy to get AUR packages into ArchVM then I take everything back and will edit my other posts that mention Plex and AUR.

 

[drum roll]

 

SUCCESS!!!  OH for F*CK Sake why isn't this little fact plainly posted?!?!

 

pacman -S yaourt
yaourt -S plexmediaserver

 

Because PMS is right now a higher version than PMS-PlexPass in AUR I had to deal with that and renaming the conf.d file but that is not different than when using pacman anyway.

 

Is the only thing I lose right now is pacman update notifications unless/until the archVM repository gets a newer version before I know it is in AUR?

 

For the Record: I was already an Arch fan, but now I'm a disciple ;-)

Link to comment

SUCCESS!!!  OH for F*CK Sake why isn't this little fact plainly posted?!?!

 

pacman -S yaourt
yaourt -S plexmediaserver

 

Because PMS is right now a higher version than PMS-PlexPass in AUR I had to deal with that and renaming the conf.d file but that is not different than when using pacman anyway.

 

You get it now. AUR kills Ubuntu because you have 100,000+ Linux God's taking 100,000+ Linux Apps and creating packages for you and they are always the latest and "tweaked" to the max. They are constantly monitored and updated when needed. You will find everything known to man in AUR. Sickrage-git for example or if you wanted to run Xen 4.5 Alpha you could install that too and not have to compile stuff yourself.

 

You still haven't learned about ABS (Arch Build System) yet.

 

Wait till all the guys here with their Ubuntu Docker Containers have to create new Docker Containers when Ubuntu 14.10 comes out (a new version every 6 months). They are going to lose the configurations for Sickbeard, CouchPotato, etc. Not to mention, EVERYTHING is going to be a nightmare when Ubuntu switches from sysinitv to systemd (next version I believe is the rumor). Basically EVERYTHING written on the web will be wrong for setting up / installing apps in Ubuntu. Plus EVERY app has to create a systemd start up (which is going to be a slow process).

 

Is the only thing I lose right now is pacman update notifications unless/until the archVM repository gets a newer version before I know it is in AUR?

 

The following will check regular Arch repos and the AUR for any updates that are available and ask you to upgrade if needed:

 

yaourt -Syyu --aur

Link to comment

Right.  Though to be fair, AUR probably has a few packages maintained by lesser men than gods ;-)  Good thing I probably won't be installing those anyway  8)

 

And ok now I see why I stopped pursuing yaourt

First you need to install package-query as a dependency, and then the yaourt package itself. Since both those packages are available from the AUR, you will have to install them with the official method for installing unsupported packages, which is exhaustively described in the AUR article.

 

That stopped me in my tracks.  I didn't bother trying to install yaourt from pacman first.

 

ABS ... yeah I've got some time on my hands right now so I'll pace myself ;)

Link to comment

Wait till all the guys here with their Ubuntu Docker Containers have to create new Docker Containers when Ubuntu 14.10 comes out (a new version every 6 months). They are going to lose the configurations for Sickbeard, CouchPotato, etc.

 

I don't think this is accurate.  As long as the new run command points to the same :/config location, the new container should see it as if nothing has changed.  I started using a ubuntu container, and am now using needo's debian based container, and it picked up the config without issue.

Link to comment

That stopped me in my tracks.  I didn't bother trying to install yaourt from pacman first.

 

The following isn't complicated for someone like you.

 

You can install yaourt from AUR:

 

curl -O https://aur.archlinux.org/packages/pa/package-query/package-query.tar.gz
tar zxvf package-query.tar.gz
cd package-query
makepkg -si
cd ..
curl -O https://aur.archlinux.org/packages/ya/yaourt/yaourt.tar.gz
tar zxvf yaourt.tar.gz
cd yaourt
makepkg -si
cd ..

 

or use our repository, add:

 

[archlinuxfr]
SigLevel = Never
Server = http://repo.archlinux.fr/$arch

to /etc/pacman.conf, and run:

 

pacman -Sy yaourt

Link to comment

I don't think this is accurate.  As long as the new run command points to the same :/config location, the new container should see it as if nothing has changed.  I started using a ubuntu container, and am now using needo's debian based container, and it picked up the config without issue.

 

You are assuming nothing changes in Ubuntu / Debian through version updates. That isn't always to be the case plus with systemd in the next the next version, I assure you... Your Debian and Ubuntu Containers will be hosed.

 

Not to mention all the other issues I laid out with using Debian based Distros in Docker:

 

The init process -- PID 1 -- inherits all orphaned child processes and must reap them. Most Docker containers do not have an init process that does this correctly. As a result their containers become filled with zombie processes over time.

 

Furthermore, docker stop sends SIGTERM to the init process, which is then supposed to stop all services. Unfortunately most init systems don't do this correctly within Docker since they're built for hardware shutdowns instead. This causes processes to be hard killed with SIGKILL, which doesn't give them a chance to correctly deinitialize things. This can cause file corruption.

 

APT incompatibilities with Docker (See https://github.com/dotcloud/docker/issues/1024)

 

syslog-ng  - A syslog daemon is necessary so that many services - including the kernel itself - can correctly log to /var/log/syslog. If no syslog daemon is running, a lot of important messages are silently swallowed. Only listens locally.

 

logrotate - Rotates and compresses logs on a regular basis.

 

ssh server - Allows you to easily login to your container to inspect or administer things.

 

cron - The cron daemon must be running for cron jobs to work.

 

Replace Upstart - Either runit or systemd. Upstart has all kinds of problems with docker and it doesn't have a service supervision or management. With runit or systemd you can restart daemons when they crash or they need to restart after an upgrade (couchpotato, sickbeard, etc). Otherwise you have to manually restart the docker container or rebooting unRAID.

Link to comment

That stopped me in my tracks.  I didn't bother trying to install yaourt from pacman first.

 

The following isn't complicated for someone like you.

 

You can install yaourt from AUR:

 

curl -O https://aur.archlinux.org/packages/pa/package-query/package-query.tar.gz
tar zxvf package-query.tar.gz
cd package-query
makepkg -si
cd ..
curl -O https://aur.archlinux.org/packages/ya/yaourt/yaourt.tar.gz
tar zxvf yaourt.tar.gz
cd yaourt
makepkg -si
cd ..

 

or use our repository, add:

 

[archlinuxfr]
SigLevel = Never
Server = http://repo.archlinux.fr/$arch

to /etc/pacman.conf, and run:

 

pacman -Sy yaourt

 

Well sure when you put it like that!  :-[  But Archwiki expected me to come up with that on my own.  And I probably could have given time.  I didn't have it last week, I have it now and for a bit.  Fortunately one of the repos I was using (IB's???) had yaourt in it and it just worked anyway.

 

Anyway, thanks.  Now, back on topic ...

Link to comment

I don't think this is accurate.  As long as the new run command points to the same :/config location, the new container should see it as if nothing has changed.  I started using a ubuntu container, and am now using needo's debian based container, and it picked up the config without issue.

 

You are assuming nothing changes in Ubuntu / Debian through version updates. That isn't always to be the case plus with systemd in the next the next version, I assure you... Your Debian and Ubuntu Containers will be hosed.

 

Not to mention all the other issues I laid out with using Debian based Distros in Docker:

 

The init process -- PID 1 -- inherits all orphaned child processes and must reap them. Most Docker containers do not have an init process that does this correctly. As a result their containers become filled with zombie processes over time.

 

Furthermore, docker stop sends SIGTERM to the init process, which is then supposed to stop all services. Unfortunately most init systems don't do this correctly within Docker since they're built for hardware shutdowns instead. This causes processes to be hard killed with SIGKILL, which doesn't give them a chance to correctly deinitialize things. This can cause file corruption.

 

APT incompatibilities with Docker (See https://github.com/dotcloud/docker/issues/1024)

 

syslog-ng  - A syslog daemon is necessary so that many services - including the kernel itself - can correctly log to /var/log/syslog. If no syslog daemon is running, a lot of important messages are silently swallowed. Only listens locally.

 

logrotate - Rotates and compresses logs on a regular basis.

 

ssh server - Allows you to easily login to your container to inspect or administer things.

 

cron - The cron daemon must be running for cron jobs to work.

 

Replace Upstart - Either runit or systemd. Upstart has all kinds of problems with docker and it doesn't have a service supervision or management. With runit or systemd you can restart daemons when they crash or they need to restart after an upgrade (couchpotato, sickbeard, etc). Otherwise you have to manually restart the docker container or rebooting unRAID.

 

Seems to me your are trying to replicate VMs with dockers. That is not the purpose of docker whatsoever. Upstart scripts should not even need to be used.

Link to comment

They (Upstart scripts) do need to be used in order to allow for the applications to update themselves or to be auto-restarted when/if they crash or exit unexpectedly.

 

Which doesn't work well in Debian based Distros and you are going to have phantom processes running and fry your file system when you stop / start Docker container enough times in a Debian Based Distro without modification (It's like yanking the power cable and not a graceful shutdown).

 

Systemd is better or a number of the other third party ones that were written for Docker.

Link to comment

They (Upstart scripts) do need to be used in order to allow for the applications to update themselves or to be auto-restarted when/if they crash or exit unexpectedly.

 

Updating an application within a Docker container via a web interface goes against the whole point of running Docker in the first place.

 

Lets say you update Sickbeard via the web interface. Then something happens and you need to regenerate your container. There goes your update, and your current database that was upgraded earlier is no longer compatible with the old version. Best case scenario SickBeard is offline, worst case scenario the database is corrupted and you have to now start from scratch.

 

Just Say No to git and updating via Web UI in Docker! :)

Link to comment

They (Upstart scripts) do need to be used in order to allow for the applications to update themselves or to be auto-restarted when/if they crash or exit unexpectedly.

 

Updating an application within a Docker container via a web interface goes against the whole point of running Docker in the first place.

 

Lets say you update Sickbeard via the web interface. Then something happens and you need to regenerate your container. There goes your update, and your current database that was upgraded earlier is no longer compatible with the old version. Best case scenario SickBeard is offline, worst case scenario the database is corrupted and you have to now start from scratch.

 

Just Say No to git and updating via Web UI in Docker! :)

 

Okay, I asked this in the other thread you posted, but again, then what is the correct option? Why wouldn't you want to update an app natively. I thought the point of moving to Docker is to get us to a point where we are not dependent on creators anymore. I would have thought I install your docker container (which is appreciated) and then I just update on my own as needed. Am I supposed to re-download your container every time an app updates itself? If so, it just seems we are tying ourselves to you, which was the basic issue with plugins, no?

 

Or maybe I am misunderstanding something (it happens).

 

Link to comment

Okay, I asked this in the other thread you posted, but again, then what is the correct option? Why wouldn't you want to update an app natively. I thought the point of moving to Docker is to get us to a point where we are not dependent on creators anymore. I would have thought I install your docker container (which is appreciated) and then I just update on my own as needed. Am I supposed to re-download your container every time an app updates itself? If so, it just seems we are tying ourselves to you, which was the basic issue with plugins, no?

 

I have not read that Dockerfiles were supposed to free any one from package maintainers. I understand the problem. And I have engineered a solution that will make everyone happy. I just haven't had time to test it out yet. Going to work on that after I convert all my Dockers to phusion.

 

I usually work from home but unfortunately have to go into the office tomorrow. Hopefully my solution will work and I will release it Wednesday/Thursday.

 

EDIT: My solution works. I'll be updating everything ASAP. This will solve this problem and make everyone happy.

Link to comment

Okay, I asked this in the other thread you posted, but again, then what is the correct option? Why wouldn't you want to update an app natively. I thought the point of moving to Docker is to get us to a point where we are not dependent on creators anymore. I would have thought I install your docker container (which is appreciated) and then I just update on my own as needed. Am I supposed to re-download your container every time an app updates itself? If so, it just seems we are tying ourselves to you, which was the basic issue with plugins, no?

 

I have not read that Dockerfiles were supposed to free any one from package maintainers. I understand the problem. And I have engineered a solution that will make everyone happy. I just haven't had time to test it out yet. Going to work on that after I convert all my Dockers to phusion.

 

I usually work from home but unfortunately have to go into the office tomorrow. Hopefully my solution will work and I will release it Wednesday/Thursday.

 

EDIT: My solution works. I'll be updating everything ASAP. This will solve this problem and make everyone happy.

 

There is no doubt everything about phusion is very clever  and this is not the first time it has been discussed.

 

However my fundamental problem with using it stems not from what we gain but what users lose.

 

Phusion is an unofficial  fork on Ubuntu. Whilst that is not an issue in itself by choosing to use it you realistically sacrifice access to the giant Ubuntu community support for all our Docker users in favor of essentially one Phusion dev.

 

 

My gut feel is that since Docker uptake is sky rocketing fixes within the real upstream repos will happen or within docker itself eventualy negating the requirement to use a base os no one has ever heard of.

 

If that sounds unreasonable then consider this. Do we expect this dev to support users until April 2019 because thats what Ubuntu have promised.

 

So would I use it, yes. Would I like it, absolutely... but I am a technically literate geek.

 

Ideally we want to support only our docks and have the base OS community support the lower OS layer and the same community plus the app devs support the upper app layer.

 

Using Phusion likely means we have no choice to support it all.

Link to comment

 

If that sounds unreasonable then consider this. Do we expect this dev to support users until April 2019 because thats what Ubuntu have promised.

 

 

Isn't the general problem here twofold:-

 

1) Debian until it moves away from upstart isn't suitable for a docker implementation yet Ubuntu is the default distribution for most people nowadays.

2) We want an easy to use package management system which implies a very stable (non cutting edge) Linux distribution. Unfortunately docker has requirements that knock out most of the stable distributions (its annoying as Ubuntu 14.04 is an LTS candidate and would otherwise be a decent core image).

 

Is it just the fact that as docker is so new, all the potential candidates have potential issues which only time will resolve and given 6 months there should be practical options. I can see why Phusion is liked but as its using runit while debian is moving towards systemd isn't it as NAS says a potential dead end...

 

Thinking things through and having slept on it I'm starting to come round to grumpybutfun's view to just use tiny core linux for the moment.

 

Unfortunately I can't join the game at the moment. I upgraded to beta 6 last night but Xen dies and as docker needs btrfs which isn't in beta 5 and I do want a working system I'll have to wait until the Xen issues are fixed before playing...

Link to comment

As for base distributions go, what are you guys so worried about, the point of docker is that we should not care about depends, diffrent versions of them or otherwise, somethings (like java based stuff) work better in 1 os rather than another. is really spending 2 Gigs of space keeping ubuntu/tcl/arch/cent bases installed Such a big deal ? (all the while complaining about the 21 drive limit ?) Let the "apps/plugins authors" do it where they feel comfortable, chanses are theyll be happier then

Link to comment

 

If that sounds unreasonable then consider this. Do we expect this dev to support users until April 2019 because thats what Ubuntu have promised.

 

Unfortunately I can't join the game at the moment. I upgraded to beta 6 last night but Xen dies and as docker needs btrfs which isn't in beta 5 and I do want a working system I'll have to wait until the Xen issues are fixed before playing...

 

Consider switching to KVM for the moment?  That way you can keep archVM, get the b6 updates, and of course have access to docker to play with.  that is what I'm probably going to try doing here in the very near future because I'm personally probably always want a VM running even if I do someday start using dicker.

Link to comment

As for base distributions go, what are you guys so worried about, the point of docker is that we should not care about depends, diffrent versions of them or otherwise, somethings (like java based stuff) work better in 1 os rather than another. is really spending 2 Gigs of space keeping ubuntu/tcl/arch/cent bases installed Such a big deal ? (all the while complaining about the 21 drive limit ?) Let the "apps/plugins authors" do it where they feel comfortable, chanses are theyll be happier then

 

Hmmm I can see it now.  Competitions to see just who has the most and most obscure base OS running on their machines  8)

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.