Installing package dependencies in order


Recommended Posts

This is an extension of this conversation: http://lime-technology.com/forum/index.php?topic=31619.msg287462#msg287462

 

Some questions:

 

In what order does unRAID install packages on boot?

Is there a sensible default?

Can we override the default?

Is there any way to actually ensure install order consistently and reliably?

 

Possible solutions:

 

1. Alphabetical by package name

2. By renaming/prefixing the package

3. By folder series, like the official Slackware remote structure

4. Some other method of dependency resolution

 

The first three are briefly discussed in the linked thread at the top of this post (it's only a few replies, worth the 60 sec read). I'd like to discuss another method as it relates to trolley and boiler (dependency resolution for 3rd party packages).

 

Boiler packages define their dependencies by name. This gets passed off to Trolley which does the actual official package resolution.

 

Packages are then saved in /boot/extra for install on reboot. Here's the problem: Say you need to run a task on install (eg, c_rehash) that depends on some package (perl in this case). The task will fail because there's no guarantee the dependency will be installed before the task needs to be run.

 

In another example, handbrake, fontconfig depends on freetype and freetype must be installed first. Package name isn't enough to ensure order.

 

One possible solution:

 

* Store boiler packages in a directory which gets install-on-reboot treatment (eg, /boot/packages/custom)

* Store packages' dependencies (official os packages) in a directory that doesn't install on reboot. (eg, /boot/system)

* Each 3rd party package will be installed as normal (natively within installpkg).

* During the post install phase, each of it's dependencies will be installed in the order they are listed.

* If two packages share a dependency, it won't be installed twice.

 

It might be useful to have a 3rd party package like "build-essential", or "development-tools" that can be used to install a baseline of deps consistently (glibc, gcc, binutils, make, etc) that are often required for higher-level applications. If it followed the method above, we could ensure dependency order. This would give great reliability to functional applications after a reboot.

 

This solutions might would work well for 3rd party packages/plugins, but it only partially addresses the native-package-order-problem outside of a plugin. If all OS packages were installed via a 3rd-party counterpart, thereby triggering the dependency tree, it might not be an issue.

 

One suggestion was to mirror the series folders, but as pointed out, that might not actual work "as is". One variation on this could be to use the package series, but don't iterate alphabetically. Instead, install the libraries series before the development series (I'm not sure if this would work in practice, but it might be possible).

 

Thoughts, ideas?

Link to comment

Been pondering this over.

 

There are two simple scenarios we need to consider as well.

 

We dont want any scheme that requires you MUST use it and therefore MUSt know it. Most packages are fine without ordered deps and we should accept this.

 

Unless you are calling a tool before all the deps are in place/loaded it doesnt matter the order they load in.

 

Does this then mean that really what we missing here is the right hooks/order in the go script? Dontknow really.

Link to comment

In what order does unRAID install packages on boot?

 

Looks like they're installed in the order found in the folder.  Here's the boot sequence from /etc/rc.d/rc.local:

# Install any extra packages
if [ -d /boot/extra ]; then
  logger "Installing /boot/extra packages"
  ( cd /boot/extra ; find -maxdepth 1 -type f -exec installpkg {} \; )
fi
shopt -s nullglob
# Install system plugins
if [ -d /boot/plugins ]; then
  logger "Installing system plugins"
  for Plugin in /boot/plugins/*.plg ; do
    /usr/local/sbin/installplg $Plugin | logger
  done
fi
# Install user plugins
if [ -d /boot/config/plugins ]; then
  logger "Installing user plugins"
  for Plugin in /boot/config/plugins/*.plg ; do
    /usr/local/sbin/installplg $Plugin | logger
  done
fi
shopt -u nullglob

# Invoke the 'go' script
if [ -f /boot/config/go ]; then
  logger "Starting go script"
  fromdos </boot/config/go >/var/tmp/go
  chmod +x /var/tmp/go
  /var/tmp/go
fi

 

Link to comment

Just to add to the topic, some time ago I did sent this to Tom as part of a PM:

 

...

 

Another thing... not related at all... what do you think about loading packages on 'extra' dir sorted by filename? That would allow us to play with filenames (i.e. add prefixes) to change packages loading order, if/when required, i.e. something like this:

( cd /boot/extra ; find -name '*.t[gblx]z' -type f -maxdepth 1 -print0 | sort -z | xargs -0 -I '{}' installpkg '{}' )

 

I did even thought about eventually something also allowing to run custom .sh scripts from that same dir,... i.e. something like:

( cd /boot/extra ; find -type f -maxdepth 1 -print0 | sort -z | while read -d $'\0' f; do   [[ "$f" =~ \.t[gblx]z$ ]] && installpkg "$f";   [[ "$f" =~ \.sh$ ]] && "$f";   done )

but not sure if you would like the idea about scripts on the 'extra' folder... the main idea would for small scripts to "setup" things related to system packages added there, load such services, etc...

 

Another idea would be an independent sort of 'go' dir... allowing a sort of multiple go-like scripts on it (and even eventually supporting some special prefixes, numbers based for eg., to define when to load each script, like pre or post emhttp, etc), think this could be maybe cleaner/easier to manage than single 'go' file. However these are just ideas you may think for the future, not something of huge importance currently.

 

At that time I was only thinking on prefixes, but now I reckon subdirs may indeed be a better idea (and replicating slack folders, if it really should work), anyway just a matter of get rid of '-maxdepth 1' and we have sorting by subdir names also. Very simple change that IMO should be at least better than having a 'random' loading order that we have now, and should be fully 'backwards compatible'.

 

Another thing to consider is 'installpkg' vs 'upgradepkg --install-new', 2nd one would make more sense, to avoid installing already stock packages (or duplicates on different folders, if that supported...) but may have one disadvantage: as some stock packages have some 'cleaned' files to save some space, the disadvantage is if user really wants to force installing the full package for some reason? though this may only be needed on more rare cases, for someone needing development tools, etc, and in this case user would surely find other ways to do it... or no one would even really need such development required files 'permanently' auto installed on startup...

Link to comment

I still dont get why this matters so much. Can someone give me a concrete common example where in the context of unraid users would be adding a function but trying to run it before all the packages are installed (or are there really daemons with deps that wont start unless the alternate packages are installed?)

 

I just worry that we make it complicated to cover an edge case that could be handled (albeit in a ugly way) right now by using the different folder order unraid already has

Link to comment

It doesn't 'matter so much' :) I even said on my quoted pm above 'not something of huge importance currently', I'm not trying to push it as a top priority thing at all! Please don't take me wrong, it isn't. Just commented it seeing an evidence of the problem (that I did mention to Tom before) with ca-certificates package. I did already faced it with some custom packages I made that would fire daemons from install script and need some libs (most stock slack packages will not do this - autostart daemons from install script, then I'm not able to give more examples, despite I wonder there may be others with issues similar to ca-certificates) and after all I just think it's just no sense loading them in a 'random' way, just that.

Link to comment

At that time I was only thinking on prefixes, but now I reckon subdirs may indeed be a better idea (and replicating slack folders, if it really should work), anyway just a matter of get rid of '-maxdepth 1' and we have sorting by subdir names also. Very simple change that IMO should be at least better than having a 'random' loading order that we have now, and should be fully 'backwards compatible'.

 

Folder series is definitely better than modifying vendor files. It would remain backwards compatible, so that's a plus. Unfortunately, it still burdens the user with doing it manually, which sucks.

 

Another thing to consider is 'installpkg' vs 'upgradepkg --install-new', 2nd one would make more sense, to avoid installing already stock packages (or duplicates on different folders, if that supported...) but may have one disadvantage: as some stock packages have some 'cleaned' files to save some space, the disadvantage is if user really wants to force installing the full package for some reason? though this may only be needed on more rare cases, for someone needing development tools, etc, and in this case user would surely find other ways to do it... or no one would even really need such development required files 'permanently' auto installed on startup...

 

I'm going to break this down because there's a lot here.

 

> Another thing to consider is 'installpkg' vs 'upgradepkg --install-new', 2nd one would make more sense, to avoid installing already stock packages (or duplicates on different folders, if that supported...)

 

`upgradepkg --install-new` is supposed to copy files from the new package (assuming same name, which isn't the case is the build or version, or anything in the filename is different) that don't already exist on the system. This sounds like a good idea at first, but it doesn't check for changes in those files, making it significantly less useful. You could actually end up with all kinds of mixed contents for the same package. `installpkg` will ensure the files get copied in full, and correctly, every time. Files are never duplicated with installpkg or upgradepkg.

 

> but may have one disadvantage: as some stock packages have some 'cleaned' files to save some space, the disadvantage is if user really wants to force installing the full package for some reason?

 

Yes, the pruning process removes some actually needed files. For example, string.h from kernel-headers, which is required for ruby to install properly. If you check to see if it's installed (exists in /var/log/packages), you'll get a false positive. If you're installing ANY software on unraid, you need the dependencies to be fully installed, else you run into very odd (and misleading) it's-installed-but-not-really problems.

 

> though this may only be needed on more rare cases, for someone needing development tools, etc

 

It isn't a rare case. Somewhere along the line, the misconception of "development tools" or "development machine" came into existence and this is only partly true. While it's true that packages like kernel-headers and gcc are classically used in development, they're also required for a lot of software to be compiled on the fly. Pretty much every *nix system has them installed by default (unraid is actually the only I've seen that doesn't). They're not large packages (largely just a compiler and a few C libraries), and they're really beneficial to have installed up front because anything that needs to be compiled (even programmatically) relies on them.

 

Folks have had issues in the past because they had the wrong versions installed (harking back to Plex not starting and the solution was to just remove all the "development stuff". The real solution is to use a more compatible version.)

 

> and in this case user would surely find other ways to do it... or no one would even really need such development required files 'permanently' auto installed on startup...

 

The idea here is to not place the burden on the user. These packages are basically required for any new packages compiled on the fly (rubygems), and they must be installed before the ruby package (this is one I have experience with, but there are likely others like this). I noticed it also happened in the Handbrake thread. There are libs that must be installed in the right order, else other things fail down the line.

 

One of the really big unreliabilities with unraid comes after a reboot, after everything is wiped out. By ensuring packages get installed in the right order, ("development" packages, or not) we can ensure higher level services can install and start more reliably. If your power goes out and you haven't touched your box in 4 months (common scenario), you don't want to spend 2 hours debugging why your services don't come back online. That's just not cool.

Link to comment

@nicinabox: with 'duplicated files' I actually meant (unnecessarily) installing package files that may be duplicated on 'extra' folder, for eg. same package at 'extra/package.txz' and at 'extra/x/package.txz', or even different package versions (already possible), didn't meant installing duplicate contents, sorry if I was not clear.  But as you pointed again, upgradepkg may cause even worst issues possibly.

Link to comment

So, there are a few different "levels" where this could be addressed.

 

1. At the Slackware level

2. At the OS level

3. At the Plugin level

4. At the User level

 

Slackware doesn't support dependency resolution at all, so that's out.

 

It *could* be done at the plugin level, albeit, it's a bit of a pain for every developer to manager that. It would be a much easier to abstract it through a tool like boiler. In that case, only boiler has to deal with the problem (as laid out in my first post).

 

And of course, we could force users to deal with the problem, pretty much like they have been. That means folks spending more time debugging their machines and spending time on the forum posting logs. That's sucky. That's unacceptable downtime, in my opinion. That's time better spent with families/friends/movies/etc.

 

So that really leaves Limetech as the best entity to address this at the OS level. I think we should consider an evolution of the /boot/{extras,packages,plugins} structure that would better accommodate dependency resolution. Better tooling could help to ease these pains.

 

There will always be a need for modularity, so this problem isn't going away unless we address it head on.

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.