• [6.8.0 RC1] encrypted unassigned drive fails to mount, docker fails to start, SMB fails to start


    BLKMGK
    • Closed Minor

     

     

    Crossposting from Reddit\6.8.0 RC1 thread - upgrade failed for me 

     

    Some additional info: I have all disks encrypted including the disk with my containers on it using Unassigned drives. This is X570 hardware, 3700X. 6.7.2 (NVIDIA) ran fine for me other than issues we had all seen. When I boot the RC I see it seem to hang as mounting drives but I can SSH in and see everything except my flash and my unassigned SSD. There are two other SSD I don't automount but I don't think they're causing issues and could pull for troubleshooting if requested. Working my way back to 6.7.2 right now but from the looks of it I can test versions freely without trashing things and am willing to do so. If you need more info holler as I'd love to help, not sure what logs are best or safe to post!

     

    Edit: am on 6.73 RC4 running fine (so far) now. Let me know how I can help!

     

    ========

    Just a heads up - this is NOT going well for my system so far. System is a TaiChi 3700X build. Docker isn't starting for some reason - NVIDIA build. I use encrypted disks but those seem to have mounted fine. Still troubleshooting and the logs look fine but no shares are showing despite mount showing a list and my being able to go into the shares via SSH. No containers are starting, VMs are down too. Screen says "mounting disks" and there it hangs. Am about to grab the standard version of this release and see if it does any better - bummer for me and I'll update as I learn more. u/sittingmongoose cross your fingers lol. Tempted to try a straight reboot first but I'll do the standard build and then try for NVIDIA. Hope I don't end up doing a parity!

    Edit: Doing a straight reboot first from the UI. Looks like it had to force shutdown - grr! coming up it says dirty and won't give the start button as it's waiting on Mounting disks  Looks like I'll drop to the standard RC1 and if that fails back to 6.7.2 NVIDIA. Noticed it's not automounting my XFS encrypted drive for containers. Tried the Mount button and it's hanging. This disk isn't showing as mounted from SSH but the others are. Maybe this is the issue?

    Edit2: fun fact, cannot download another unRAID version to my USB while things won't mount. I may have to pull this from the rack. I'll try SFTP and hope I have a handy backup around  SFTP gets me in at least but my share for Flash isn't showing dammit.

    Edit3: Normal unRAID upgrade tool had a backup version of 6.7.0 RC6 in it, guess it's been awhile since I used that  Managed to boot it! For science sake I'll try the normal unRAID 6.8 RC1 and see if it works better using that tool. If not I at least have a working go-back and we'll know if this was NVIDIA or not.

    Edit4: nope same issue with normal 6.8 RC1 for me. It looks like my use of encrypted disks could be an issue? Possibly because I have an encrypted secondary disk using Unassigned devices - it's not clear to me but it looks like I can get back to 6.7.0 RC6. I will then use normal tool to get to 5.7.2 and then NVIDIA 6.7.2. Bummer, I really wanted this to work 

    This is what I see when I try to manually mount my Lux encrypted container disk:

    Oct 14 01:28:59 Minion unassigned.devices: Adding disk '/dev/mapper/docker_vm'...

    Oct 14 01:28:59 Minion unassigned.devices: luksOpen: key file not found - using emcmd to mount.

    minion-diagnostics-20191014-0132.zip




    User Feedback

    Recommended Comments



    The array appears to be waiting for the password.  Unassigned devices can't mount the device because there is no password.

     

    I don't use encrypted disks, but I believe that you need to enter the password before the array will start.

    Link to comment
    3 hours ago, bonienl said:

    Do you use a passphrase or (binary) keyfile for encryption?

    I use a passphrase that's entered at each boot, I am entering the passphrase in order to start the array. I am able to SSH in and see the contents of my protected drives without issue but SMB sharing is apparently not starting and I cannot see those shares from the network. I cannot see the unassigned drive as it never seems to mount. Under 6.7.2 and 6.7.3 rc4 this works.

     

     

    Now that it's not 1am pushing 4:30 am some clarity lol. My array consists or multiple drives that are all encrypted. My cache drive is also encrypted, my single drive outside of the array used for VM and container storage is also encrypted. When my system boots it requests a passcode before proceeding to mount drives and the Start button won't light without an entry. In the failure case I type in the password, the button lights, I hit start, and the interface eventually refreshes to show my drives up (no more dropdowns) but my Unassigned Drive outside of the array is failing to automount. As this is where VM and Container files are held this is "bad". When this occurs it appears that the normal sharing also fails and the rest of the array doesn't start.

     

    For those wondering - this encrypted drive outside of the array was created by temporarily making it my cache drive, formatting and encrypting it, and then returning my original encrypted cache drive to use. I then mount this drive like any other at boot with automount using Unassigned Drives. My entire array is encrypted as a result and has been since the feature was introduced (I lobbied for it for years). The drive outside the array was only recently encrypted, my thanks to u/Spaceinvaderone for the how-to on that trick just as I was upgrading cache and Container drives  :D  I could reverse this and decrypt that outside drive but it stores data I'd prefer to remain encrypted and it's worked well through multiple versions prior to this.

     

    Questions? Fire away, I'd love to solve this! This might be Unassigned Drives that needs a tweak, I'll hunt down the support thread for that as well, but something has most certainly changed somewhere in this "release candidate". So just to be clear I knew there was a risk here, am NOT upset, and simply want to help get to the bottom of this as there's a TON of fixes in this I'd love to obtain :D Much thanks to the NVIDIA guys for compiling against it too!

    Link to comment

    I am guessing that this problem is caused by the following change from the 

    release notes (which was made to increase security):

    Quote

    Encryption - an entered passphrase is not saved to any file.  Also included an API for Unassigned devices plugin to open encrypted volumes.

    This probably means that Unassigned Devices can no longer find the pass phrase and is going to need some rework before in can support encrypted disks again using the API mentioned

    • Like 1
    Link to comment
    34 minutes ago, itimpi said:

    I am guessing that this problem is caused by the following change from the 

    release notes (which was made to increase security):

    This probably means that Unassigned Devices can no longer find the pass phrase and is going to need some rework before in can support encrypted disks again using the API mentioned

    Aw crap, I missed that! Well okay, that seems to have solved the mystery and explains the change in behaviour. Working as designed and anticipated it seems - d'oh! I suppose for now I can either save to a file or decrypt the drive, not sure which is yuckier lol

    Link to comment
    2 minutes ago, limetech said:

    That would be a good test to confirm the issue.

    Ah ha, I linked to root a keyfile stored on my USB (shiver) in the Go file and it's booted! I'm getting some ugliness in the logs though as it seems something is crashing. From the UI all appears normal for now and I'll let it roll. Appreciate the assist!


    Here's a link to a snippet of what I see in my logs currently and it began right after a container fired up (it's running fine).

    https://pastebin.com/HtKwsfx4

    Link to comment
    Just now, BLKMGK said:

    Here's a link to a snippet of what I see in my logs currently and it began right after a container fired up (it's running fine).

     

    • Like 1
    Link to comment

    Thank you @Squid that looks to be exactly what's going on. I run both VM and multiple containers. I'll keep an eye on the log size! Happy to not be alone :D 

    Link to comment
    5 hours ago, itimpi said:

    I am guessing that this problem is caused by the following change from the 

    release notes (which was made to increase security):

    This probably means that Unassigned Devices can no longer find the pass phrase and is going to need some rework before in can support encrypted disks again using the API mentioned

    Unassigned Devices has been updated to deal with this change.

    Quote

    Oct 14 01:28:59 Minion unassigned.devices: Adding disk '/dev/mapper/docker_vm'...

    Oct 14 01:28:59 Minion unassigned.devices: luksOpen: key file not found - using emcmd to mount.

    This means UD didn't find the key file and is using the new API.

     

    I do not have an encrypted array and could only do limited testing.  Not sure if this is a UD issue or the API.  There are no entries in the log to indicate what went wrong.

    Link to comment

    This is what the log should show when the keyfile is not found:

    Oct 14 18:22:19 BackupServer unassigned.devices: luksOpen: key file not found - using emcmd to mount.
    Oct 14 18:22:19 BackupServer emhttpd: req (7): cmdCryptsetup=luksOpen /dev/sdg1 Test_Disk --allow-discards&csrf_token=****************
    Oct 14 18:22:19 BackupServer unassigned.devices: Mount drive command: /sbin/mount '/dev/mapper/Test_Disk' '/mnt/disks/Test_Disk'
    Oct 14 18:22:19 BackupServer unassigned.devices: Mount of '/dev/mapper/Test_Disk' failed. Error message: mount: /mnt/disks/Test_Disk: special device /dev/mapper/Test_Disk does not exist.
    Oct 14 18:22:19 BackupServer unassigned.devices: Partition 'WDC_WD3200AAJS-65M0A0_WD-WMAV21954758' could not be mounted...
    O

    I don't have an encrypted array and there is no password in my system.  I would expect the mount to fail.  What I don't understand is I don't see the emhttp command being executed in the log file.  It should try to mount the disk after the emhttp command is executed, or print an error if the command returned an error.  I see neither.

     

    The difference here is that your array is all encrypted.  Mine is not.

    Link to comment
    2 hours ago, dlandon said:

    The difference here is that your array is all encrypted.  Mine is not.

    Maybe he's running less-than-latest version of UD?

    Link to comment
    35 minutes ago, limetech said:

    Maybe he's running less-than-latest version of UD?

    He's running the latest.  Why would emcmd not run?  Or not appear to run?

     

    Edit: The emcmd is not running and is not returning to UD.  UD will either attempt to mount the disk or report an error after the emcmd command.  Neither of those is happening.

    Edited by dlandon
    Link to comment
    3 hours ago, dlandon said:

    He's running the latest.  Why would emcmd not run?  Or not appear to run?

    Works for me.  Example:

    emcmd "cmdCryptsetup=luksOpen /dev/sdg1 Test_Disk --allow-discards"

    results in these syslog lines:

    Oct 14 21:24:08 Test1 emhttpd: req (14): cmdCryptsetup=luksOpen /dev/sdg1 Test_Disk --allow-discards&csrf_token=****************
    Oct 14 21:24:08 Test1 emhttpd: shcmd (5882): /usr/sbin/cryptsetup luksOpen /dev/sdg1 Test_Disk --allow-discards --key-file=-

    The operation fails because on this test server there's no 'sdg1'.  But if I mount an md-device for an encrypted volume where the passphrase has already been entered:

    emcmd "cmdCryptsetup=luksOpen /dev/md4 md4"

    syslog lines are similar but /dev/mapper/md4 now exists.  I should add some code to print non-success exit status to syslog but the call works.

    Link to comment
    5 hours ago, limetech said:

    I should add some code to print non-success exit status to syslog but the call works.

    Return the results of the luksOpen command and I'll log it.  I can take appropriate action based on the results.  If it fails, I don't want to attempt a mount.

     

    I'll add some debug so we can see better what is happening.

    Link to comment

    Hi, I have updated to the newiest version of Unassigned devices and on RC1

     

    When it trys to map my unassigned devices it does the following but then does not do anything, the array does not finish mounting so none of my services start.

     

    Oct 15 19:17:36 Tower unassigned.devices: Mounting 'Auto Mount' Devices...
    Oct 15 19:17:36 Tower unassigned.devices: Adding disk '/dev/mapper/Crucial_1TB'...
    Oct 15 19:17:36 Tower unassigned.devices: luksOpen: key file not found - using emcmd to open: 'cmdCryptsetup=luksOpen /dev/sdk1 Crucial_1TB --allow-discards'

    Link to comment
    4 hours ago, dan91 said:

    Hi, I have updated to the newiest version of Unassigned devices and on RC1

    Did you see the post immediately above yours?

    Link to comment
    5 hours ago, dan91 said:

    Hi, I have updated to the newiest version of Unassigned devices and on RC1

     

    When it trys to map my unassigned devices it does the following but then does not do anything, the array does not finish mounting so none of my services start.

     

    Oct 15 19:17:36 Tower unassigned.devices: Mounting 'Auto Mount' Devices...
    Oct 15 19:17:36 Tower unassigned.devices: Adding disk '/dev/mapper/Crucial_1TB'...
    Oct 15 19:17:36 Tower unassigned.devices: luksOpen: key file not found - using emcmd to open: 'cmdCryptsetup=luksOpen /dev/sdk1 Crucial_1TB --allow-discards'

    Post diagnostics.  There will be more in the log after that.

    Link to comment

    I have updated UD and will attempt to test tonight. Just so I'm clear, you want me to remove the link fix I've got now, allow it to fail, grab the diagnostic, and pop it up here? I will have to drop back to the old 6.7 RC that's weirdly onboard and then upgrade again to recover so I'm hoping these are the needed steps heh. Server is a touch busy right now but I'll try to do it ASAP so if there's something special you need and you're online drop it here and I'll hopefully spot it before doing this.

     

    P.S. Tom, could you guys please compile in the Overlay module for Docker going forward? I think it's the only thing preventing me from using Swarm with my server :D

    minion-diagnostics-20191016-0001.zip

    Edited by BLKMGK
    Adding requested log files!
    Link to comment
    7 hours ago, limetech said:

    Did you see the post immediately above yours?

     

    7 hours ago, dlandon said:

    Post diagnostics.  There will be more in the log after that.

     

    Hi please see diagnostics attached but that is as far as it gets it just hangs.

     

    Would it be anything to do with i'm using the second password on my LUKS volume due to the post I made in the main thread?

     

     

    tower-diagnostics-20191016-0642.zip

    Edited by dan91
    Link to comment

    Be sure the UD disk is unmounted and then open a terminal window and enter:

    /usr/local/sbin/emcmd 'cmdCryptsetup=luksOpen /dev/sdk1 Crucial_1TB --allow-discards'

    Post the results of the command.

    Link to comment
    8 hours ago, dlandon said:

    Be sure the UD disk is unmounted and then open a terminal window and enter:

    
    /usr/local/sbin/emcmd 'cmdCryptsetup=luksOpen /dev/sdk1 Crucial_1TB --allow-discards'

    Post the results of the command.

    Hi,

     

    It doesnt do anything unfortunately it seems to just hang.

     

    image.png.eee89a179d5f45fb8825910acb5d0114.png

     

     

    tower-diagnostics-20191016-1851.zip

    Link to comment
    10 hours ago, dan91 said:

    It doesnt do anything unfortunately it seems to just hang.

    There's something wrong with handling of ridiculously wrong long passphrases.  We'll look into it.

    Edited by limetech
    2 wrongs make a long
    • Haha 1
    Link to comment
    40 minutes ago, limetech said:

    There's something wrong with handling of ridiculously wrong passphrases.  We'll look into it.

    Wrong or Long? 🤣😂

    After the problem I had with the 512 char limit ive shortened it now to under 260!

     

    Link to comment



    Join the conversation

    You can post now and register later. If you have an account, sign in now to post with your account.
    Note: Your post will require moderator approval before it will be visible.

    Guest
    Add a comment...

    ×   Pasted as rich text.   Restore formatting

      Only 75 emoji are allowed.

    ×   Your link has been automatically embedded.   Display as a link instead

    ×   Your previous content has been restored.   Clear editor

    ×   You cannot paste images directly. Upload or insert images from URL.


  • Status Definitions

     

    Open = Under consideration.

     

    Solved = The issue has been resolved.

     

    Solved version = The issue has been resolved in the indicated release version.

     

    Closed = Feedback or opinion better posted on our forum for discussion. Also for reports we cannot reproduce or need more information. In this case just add a comment and we will review it again.

     

    Retest = Please retest in latest release.


    Priority Definitions

     

    Minor = Something not working correctly.

     

    Urgent = Server crash, data loss, or other showstopper.

     

    Annoyance = Doesn't affect functionality but should be fixed.

     

    Other = Announcement or other non-issue.