• [6.12.3] /usr, /lib unmounting results in unavailable binaries for clean shutdown


    Rysz
    • Solved Urgent

    In previous versions of UNRAID the /usr and part of the /lib folder were a part of the root filesystem / and in the process were never unmounted because the root filesystem / persisted in RAM throughout the shutdown sequence in /etc/rc.d/rc.6. This also meant that all the binaries in /usr/sbin and /usr/local/sbin were available to the services until the system halted entirely, ensuring - for example - plugins could also run their respective shutdown sequences gracefully by utilizing their binaries in /usr/sbin (e.g. shutdown devices, stop their associated services etc.)

     

    In newer versions of UNRAID (e.g. the latest 6.12.3) the /usr and entire /lib folder are no longer part of the root filesystem / and are unmounted being separate mount-points due to the mass-unmount-all command present early on in /etc/rc.d/rc.6:

     

    # Unmount local file systems:
    echo "Unmounting local file systems:"
    /bin/umount -v -a

     

    From that point on anything in the /usr and /lib folders becomes unavailable including anything in /usr/sbin or /usr/local/sbin, which is curious because the shutdown script actually proceeds to reference such an unavailable binary directly below that mass-unmount-all command:

     

    # Unmount local file systems:
    echo "Unmounting local file systems:"
    /bin/umount -v -a
    
    # limetech - shut down the unraid driver if started
    if /bin/grep -qs 'mdState=STARTED' /proc/mdstat ; then
      echo "Stopping md/unraid driver:"
      /usr/local/sbin/mdcmd stop
      if ! /bin/grep -qs 'mdState=STOPPED' /proc/mdstat ; then
        echo "Unclean shutdown - Cannot stop md/unraid driver"
      else
        # we have to mount /boot again
        if ! /sbin/mount -v /boot ; then
          echo "Unclean shutdown - Cannot remount /boot"
        else
          /bin/rm -f /boot/config/forcesync
          /sbin/umount /boot
          echo "Clean shutdown"
        fi
      fi
    fi

     

    This if-clause will never be able to stop the md/unraid driver (if it is started), because the binary mdcmd at that point is already unmounted and unavailable due to the command above that if-close unmounting the entire /usr tree.

     

    Another problem we're facing with this behaviour is - as an example - our NUT plugin (for UPS) being unable to shutdown the UPS inverter because the binary call to /usr/sbin/upsdrvctl is made impossible by the premature unmounting of /usr - see further details here:

     

    I therefore propose the developers consider persisting the /usr and /lib mount-points until the system halts, as they should only be in RAM anyhow and their unmounting seems to be no requirement for a graceful shutdown but rather complicates such a graceful shutdown for both the plugins and the core system itself.

     

    Perhaps reverting to either specifying which types of filesystems to unmount or which types of filesystems not to unmount utilizing umount -v -a -t instead of just umount -v -a could be considered, as it was done in earlier versions of UNRAID.

     

    Overall, is unmounting the local filesystems on shutdown even necessary at all when they exist just in RAM?

     

    • Thanks 1
    • Upvote 1



    User Feedback

    Recommended Comments

    Thank you for the report, yes indeed a bug here.

     

    Quote

    our NUT plugin (for UPS) being unable to shutdown the UPS inverter because the binary call to /usr/sbin/upsdrvctl is made impossible by the premature unmounting of /usr

     

    From where is the call to this executable made, is it via /sbin/genpowerd ?

    Link to comment
    28 minutes ago, limetech said:

    Thank you for the report, yes indeed a bug here.

     

     

    From where is the call to this executable made, is it via /sbin/genpowerd ?

     

    Thanks for checking, the NUT plugin inserts the UPS inverter shutdown above the "# Now halt [...]" comment:

     

    [ -x /etc/rc.d/rc.nut ] && /etc/rc.d/rc.nut shutdown
    # Now halt (poweroff with APM or ACPI enabled kernels) or reboot.
    if [ "$shutdown_command" = "reboot" ]; then
      echo "Rebooting."
      /sbin/reboot
    else
      /sbin/poweroff
    fi

     

    The script then in turn calls the binary /usr/sbin/upsdrvctl shutdown (however /usr/sbin at that stage is no longer available due to unmounting) to power off the UPS inverter.

     

    So basically I think the local (RAM) filesystems should be available until the system halts entirely, as in older versions where this still works. It shouldn't have a negative effect on a graceful shutdown to keep those filesystems mounted since all in RAM is lost on reboot regardless, if I'm not misunderstanding something here.

     

    The upside of keeping the filesystems mounted would be that all services (be it plugins or other core processes) could do everything to facilitate a clean shutdown right until the very end (system halt) without having any of their resources taken away beforehand. 🙂

    Edited by Rysz
    • Thanks 1
    Link to comment

    There is a problem: during shutdown we definitely want to cleanly unmount the usb flash which is mounted at /boot.  However, in order to do that we must first unmount the two squashfs file systems: /usr and /lib.

     

    Does that 'rc.nut shutdown' operation kill power immediately?  If so, any executable it uses must get moved off of /usr, maybe put into /bin.

    Link to comment
    13 minutes ago, limetech said:

    There is a problem: during shutdown we definitely want to cleanly unmount the usb flash which is mounted at /boot.  However, in order to do that we must first unmount the two squashfs file systems: /usr and /lib.

     

    Does that 'rc.nut shutdown' operation kill power immediately?  If so, any executable it uses must get moved off of /usr, maybe put into /bin.

     

    What about other plugins, would they need to hack around these limitations too? If so, then it seems like maybe instead of having every single plugin attempt to work around this, having the system itself change is the more efficient approach? 🤷‍♂️

    Link to comment
    16 minutes ago, BRiT said:

    What about other plugins, would they need to hack around these limitations too?

     

    If they are involved in shutting down the server, yes.

    Link to comment

    Wouldn't this work, for example?

     

    /bin/umount -v -a -t no,proc,sysfs,devtmpfs,fuse.gvfsd-fuse,tmpfs,overlay

     

    That way everything not on the "no,[...]" list would get unmounted cleanly, including the vfat /boot filesystem.

    Since /boot is a vfat filesystem we could unmount it this way while also keeping our local filesystems mounted.

     

    This was actually already done cleanly (unmounting everything but local filesystems) in 6.8.3 for example:

     

    # Unmount local file systems:
    # limetech - remove /boot, /lib/firmware, and /lib/modules from mtab first
    /bin/umount --fake /boot
    /bin/umount --fake /lib/firmware
    /bin/umount --fake /lib/modules
    
    echo "Unmounting local file systems:"
    /bin/umount -v -a -t no,proc,sysfs,devtmpfs,fuse.gvfsd-fuse,tmpfs
    
    # limetech - shut down the unraid driver if started
    if grep -qs 'mdState=STARTED' /proc/mdstat ; then
      echo "Stopping md/unraid driver:"
      /usr/local/sbin/mdcmd stop
    fi
    # limetech - now unmount /lib/firmware, /lib/modules and /boot
    /bin/umount -v /lib/firmware
    /bin/umount -v /lib/modules
    /bin/umount -v /boot

     

    We'd just have to account for the new "overlay" filesystem to add to the "no,[...]" list of local filesystems to persist.

     

    Edited by Rysz
    Link to comment

    embed UPS solution (apcupsd) also does not work properly (do not turn of UPS) after update to 6.12

    • Thanks 1
    Link to comment
    13 hours ago, nau said:

    embed UPS solution (apcupsd) also does not work properly (do not turn of UPS) after update to 6.12

     

    I just ran a test in 6.12.3 and apcupsd stopped the array and shut down the server without issue when the "Runtime left to initiate shutdown" condition was met.
     

    In terms of turning off the UPS, it depends on whether your UPS supports the APC command for that or not. I have a Cyberpower UPS and it does not. For more about this, see https://forums.unraid.net/topic/38606-apcupsd-settings-does-not-seem-to-shut-down-ups-solved/#comment-375297 

     

    Link to comment
    14 hours ago, nau said:

    It is indeed supported by my UPS and worked perfectly fine for years just before update to 6.12.

     

    Interesting. I don't see a way to split this conversation into a new bug report, so please create a separate bug report for this.  The more info you can provide the better. Such as, what is the make/model of the UPS, what was the last version that shut it down, etc.

    Link to comment

    Hey folks! We have a 6.12.4 rc release available that should allow the NUT plugin to shut the server down, would appreciate your help confirming it resolves this issue:

     

    • Like 1
    • Thanks 1
    Link to comment
    17 hours ago, ljm42 said:

    Hey folks! We have a 6.12.4 rc release available that should allow the NUT plugin to shut the server down, would appreciate your help confirming it resolves this issue:

     

    thank you very much! This release fixed problem with apcupsd. Now UPS is being shutting down and start after.

    • Like 1
    • Thanks 1
    Link to comment

    Thanks a lot - I can confirm this working with NUT on my instance.

    All relevant binaries and drivers remain accessible throughout the shutdown process.

     

    I've directed the other users experiencing this problem to the RC version as well.

    Hopefully we'll have one or two more people confirming this as working as expected now.

    • Thanks 1
    Link to comment


    Join the conversation

    You can post now and register later. If you have an account, sign in now to post with your account.
    Note: Your post will require moderator approval before it will be visible.

    Guest
    Add a comment...

    ×   Pasted as rich text.   Restore formatting

      Only 75 emoji are allowed.

    ×   Your link has been automatically embedded.   Display as a link instead

    ×   Your previous content has been restored.   Clear editor

    ×   You cannot paste images directly. Upload or insert images from URL.


  • Status Definitions

     

    Open = Under consideration.

     

    Solved = The issue has been resolved.

     

    Solved version = The issue has been resolved in the indicated release version.

     

    Closed = Feedback or opinion better posted on our forum for discussion. Also for reports we cannot reproduce or need more information. In this case just add a comment and we will review it again.

     

    Retest = Please retest in latest release.


    Priority Definitions

     

    Minor = Something not working correctly.

     

    Urgent = Server crash, data loss, or other showstopper.

     

    Annoyance = Doesn't affect functionality but should be fixed.

     

    Other = Announcement or other non-issue.