• [6.8.0 RC1] encrypted unassigned drive fails to mount, docker fails to start, SMB fails to start


    BLKMGK
    • Closed Minor

     

     

    Crossposting from Reddit\6.8.0 RC1 thread - upgrade failed for me 

     

    Some additional info: I have all disks encrypted including the disk with my containers on it using Unassigned drives. This is X570 hardware, 3700X. 6.7.2 (NVIDIA) ran fine for me other than issues we had all seen. When I boot the RC I see it seem to hang as mounting drives but I can SSH in and see everything except my flash and my unassigned SSD. There are two other SSD I don't automount but I don't think they're causing issues and could pull for troubleshooting if requested. Working my way back to 6.7.2 right now but from the looks of it I can test versions freely without trashing things and am willing to do so. If you need more info holler as I'd love to help, not sure what logs are best or safe to post!

     

    Edit: am on 6.73 RC4 running fine (so far) now. Let me know how I can help!

     

    ========

    Just a heads up - this is NOT going well for my system so far. System is a TaiChi 3700X build. Docker isn't starting for some reason - NVIDIA build. I use encrypted disks but those seem to have mounted fine. Still troubleshooting and the logs look fine but no shares are showing despite mount showing a list and my being able to go into the shares via SSH. No containers are starting, VMs are down too. Screen says "mounting disks" and there it hangs. Am about to grab the standard version of this release and see if it does any better - bummer for me and I'll update as I learn more. u/sittingmongoose cross your fingers lol. Tempted to try a straight reboot first but I'll do the standard build and then try for NVIDIA. Hope I don't end up doing a parity!

    Edit: Doing a straight reboot first from the UI. Looks like it had to force shutdown - grr! coming up it says dirty and won't give the start button as it's waiting on Mounting disks  Looks like I'll drop to the standard RC1 and if that fails back to 6.7.2 NVIDIA. Noticed it's not automounting my XFS encrypted drive for containers. Tried the Mount button and it's hanging. This disk isn't showing as mounted from SSH but the others are. Maybe this is the issue?

    Edit2: fun fact, cannot download another unRAID version to my USB while things won't mount. I may have to pull this from the rack. I'll try SFTP and hope I have a handy backup around  SFTP gets me in at least but my share for Flash isn't showing dammit.

    Edit3: Normal unRAID upgrade tool had a backup version of 6.7.0 RC6 in it, guess it's been awhile since I used that  Managed to boot it! For science sake I'll try the normal unRAID 6.8 RC1 and see if it works better using that tool. If not I at least have a working go-back and we'll know if this was NVIDIA or not.

    Edit4: nope same issue with normal 6.8 RC1 for me. It looks like my use of encrypted disks could be an issue? Possibly because I have an encrypted secondary disk using Unassigned devices - it's not clear to me but it looks like I can get back to 6.7.0 RC6. I will then use normal tool to get to 5.7.2 and then NVIDIA 6.7.2. Bummer, I really wanted this to work 

    This is what I see when I try to manually mount my Lux encrypted container disk:

    Oct 14 01:28:59 Minion unassigned.devices: Adding disk '/dev/mapper/docker_vm'...

    Oct 14 01:28:59 Minion unassigned.devices: luksOpen: key file not found - using emcmd to mount.

    minion-diagnostics-20191014-0132.zip




    User Feedback

    Recommended Comments



    8 hours ago, dlandon said:

    Read several posts up.  LT has noted an issue with long passwords.  Have you tried rc3?

    RC3 hasn't been installed here yet but I'll try to do so today and report back. My password is under 25 chars in length.

     

    Edit: RC3 has the same behavior as RC1 with the added "benefit" that I can no longer go back to a working version of unRAID via the GUI. I've been able to edit the Go file from /boot/config to add my link file back to work around this for now. A parity check was triggered as well.

     

    Manual use of the emcmd against my drive resulted in a hang same as the other user, no output provided at the console from this command.

    minion-diagnostics-20191020-1613.zip

    Edited by BLKMGK
    Link to comment
    19 hours ago, BLKMGK said:

    @dlandon Is RC4 supposed to have anything to help sort this?

     

    No.  Limetech will ask you to re-test when there is a release with a fix.

    Link to comment

    The "fix" for this issue is: If you are using UD plugin with encrypted volumes, please use the 'file' method of specifying the encryption key, taking care to exclude any line ending characters when you put your passphrase into a file.

    Link to comment
    17 hours ago, limetech said:

    The "fix" for this issue is: If you are using UD plugin with encrypted volumes, please use the 'file' method of specifying the encryption key, taking care to exclude any line ending characters when you put your passphrase into a file.

    How exactly is this a "fix"?! I'm using this method now and it requires me to place a cleartext password on my drive where it's easily accessible - this defeats the encryption completely! Frankly I liked it better where a temporary file was created because at least then it would be ephemeral if power was removed. Am I somehow misunderstanding you here?

     

    If I'm understanding you correctly this completely defeats the purpose of encryption the drives - please reconsider!

    Link to comment

    You would need to store your passphrase in a file somewhere else, e.g. your PC.

    When starting the array then choose the keyfile method and select the file you have created above.

    This approach allows UD to work properly.

     

    Disclosure: Unraid 6.9 will support multiple pools, and allows pool assignments to specific VMs and/or docker. Effectively taking over some functionality of UD.

     

     

    Edited by bonienl
    Link to comment
    13 hours ago, bonienl said:

    You would need to store your passphrase in a file somewhere else, e.g. your PC.

    When starting the array then choose the keyfile method and select the file you have created above.

    This approach allows UD to work properly.

     

    Disclosure: Unraid 6.9 will support multiple pools, and allows pool assignments to specific VMs and/or docker. Effectively taking over some functionality of UD.

     

     

     

    So if I'm remote and need to restart my computer from my phone or tablet - not a PC - I may not be able to do so? It would seem we've just lost functionality here. I'm often far away from home when I'm forced to reboot or need to enter a password to decrypt and I do it from my phone or tablet - IOS.

    Storing a cleartext password on my PC vs storing it ephemerally on my USB stick (with an option to remove) appears to be less secure. As it stands now I can use a phrase or pull a password from a secured app. I've seen where others have tried to use image files and things for a password file and perhaps that's better were my password not already set.

     

    How will multiple pools change this situation? If you're saying that we'll be able to have drives outside of the array for VMs and Containers that's terrific - will we be able to encrypt them?

    Link to comment

    UD was originally designed and implemented to allow USB devices to be plugged into Unraid and files transferred on and off the array with ease.  It then morphed into mounting disks for VMs and Dockers and remote mounting SMB/NFS shares.  I use a UD disk for doing some daily backups and mounting shares on a remote Unraid server.

     

    It was always intended that VMs and Dockers would be on the array or the cache drive.  Using UD to mount disks for VMs and Dockers works fine, even though it was not intended for that purpose.

     

    Then it was requested that an encrypted drive be mountable by UD so an array drive cold be mounted and the files taken off the drive.  Mounting an encrypted drive by UD was never intended to be operated as VM or Docker storage.  That's why you have run into the situation that you have now that LT is tightening security.

     

    LT is responding to the need you have by implementing cache pools.  UD needs to go back to what was originally intended - mount USB disks for file transfer, mount hard drives for backups, temporary storage, remote SMB/NFS mounts, etc and not a part of array, VM, or Docker operation.

     

    Lt does not intend to do anything further to work out the password situation with UD devices.  I don't intend to do anything further with UD to handle the password situation.  LT is working on implementing cache pools that is the correct solution to your situation and is the best solution in the long run.

     

    You have several options:

    - Continue to use it as you have it set up now with the recognized inconvenience of using the password file.

    - Use the UD device without encryption.

    - Put your Docker image on the array or cache drive where it belongs.

    Edited by dlandon
    Link to comment
    6 hours ago, BLKMGK said:

    If you're saying that we'll be able to have drives outside of the array for VMs and Containers that's terrific - will we be able to encrypt them?

    Yes, and yes.  The drives will appear like multiple cache disks.

    Link to comment

    I originally moved my VMs and Containers off of the cache because the size of the storage wasn't enough for moving large files - not without spending a pile. This was of particular importance when running the mover began tanking performance of the entire system. Like many I move video files around and those can get BIG. Run out of space on the Cache drive while moving files around and things come to a halt. I needed as much space for actual cache as possible! I currently move files nightly but can perhaps stop that now that mover doesn't kill things. If I can use more than one drive for cache and keep it encrypted then sure, that solves the issue and hopefully moving things over will be easy. Bear in mind that if you run Plex with a decent library you're talking over a million (tiny) files for metadata. My Containers and VMs take up over 240Gig right now so yeah, I moved them off my cache drive and I'm betting others have too. Putting this on the array is a non-starter, the pause when a sleeping drive spins up and general performance of spinning media makes that clear. I didn't move my files on a whim and I encrypt them to deny a thief, get your stuff stolen a time or two and you get angry like that. Look back and guess who it was that asked for crypto to be added years ago :D

     

    As for backing stuff up - I'd want that encrypted too! Why bother encrypting otherwise? If I'm pulling sensitive data off for safe keeping I'd like it safe. Currently my personal backup is cloud storage and that's heavily encrypted too. Being able to plug in USB for transfer, connect to other datastores, and things I've never thought of though are awesome features but I've not been using those a much.

     

    Let me clear up how I'm currently forced to run my system. I've got a cleartext file on my USB containing my password. I have a line in my GO script that creates a link to it in the ephemeral filesystem on boot. My system currently boots hands-off and is completely insecure this way. I've left it this way to continue testing the RC. I've seen some weirdness with UD dropping drives that I've attributed to controller issues and my logs filling with crap, but otherwise it's been stable overall. I still badly need Swarm support though!

     

    I've never tried uploading a file to boot my system, doing it on mobile might not even be possible. Using a file like a JPEG or whatever hadn't occurred to me until I saw someone else mention it. Switching to something like that at least makes it less obvious what cleartext file on my system is the password but mobile is still an issue. I wonder if the browser will try to remember my file selection? I have another server I can play with so perhaps I'll use that to work things out instead of risking my primary.

     

    I'll move to the larger cache "pool" when it's available if that keeps things secured but it's sounding like that's not exactly around the corner.

    Link to comment
    14 minutes ago, BLKMGK said:

    Let me clear up how I'm currently forced to run my system.

    I feel your pain.  Here's what I think we can do.  I'll create a new config setting, call it "BLKMGK mode", when set to Yes then when emhttpd initiates the mount sequence it will write the plaintext passphrase to /root/keyfile and then at end of mount sequence it will delete the file.  That way UD can pick up the passphrase - but bear in mind, any other plugin could possibly pick it up too.  Sound good?

    • Like 1
    Link to comment
    2 hours ago, limetech said:

    I feel your pain.  Here's what I think we can do.  I'll create a new config setting, call it "BLKMGK mode", when set to Yes then when emhttpd initiates the mount sequence it will write the plaintext passphrase to /root/keyfile and then at end of mount sequence it will delete the file.  That way UD can pick up the passphrase - but bear in mind, any other plugin could possibly pick it up too.  Sound good?

    The problem with this is that if the disk is unmounted, the keyfile will not be there to re-mount.  Or if an encrypted disk is added after the array is mounted.  The keyfile will be missing.

     

    Link to comment
    8 hours ago, limetech said:

    I feel your pain.  Here's what I think we can do.  I'll create a new config setting, call it "BLKMGK mode", when set to Yes then when emhttpd initiates the mount sequence it will write the plaintext passphrase to /root/keyfile and then at end of mount sequence it will delete the file.  That way UD can pick up the passphrase - but bear in mind, any other plugin could possibly pick it up too.  Sound good?

    Would that allow for the entry of a text passphrase? In essence that undoes the feature you've implemented but with the added benefit of deleting the file. I'm okay with that risk! I recognize other plug-ins could snatch it and that's a good catch to think of but I can't be but so paranoid 😮 

     

    5 hours ago, dlandon said:

    The problem with this is that if the disk is unmounted, the keyfile will not be there to re-mount.  Or if an encrypted disk is added after the array is mounted.  The keyfile will be missing.

     

    You bring up a good point, I guess I'd ask - is this taking an edge case and extending it? Alternately, would it be possible to re-prompt for the passphrase when this occurs? I don't know what control you get when a drive mounts to fire a prompt. It occurs to me that if you could do that you could even use a different password for the mount maybe? This would allow for your "USB storage for backups to be attached "use case and also allow encryption. Not sure it's doable but maybe a solution?

    I mentioned my drive occasionally dropping out on my mobo controller, I've since moved it. In this case the issue you've pointed out would trip me up but when it's occurred I've always shut down in order to restart my containers and VMs so in a sense it's already hit me. The question would be - would this impact anyone else? So far I seem to be the only one whining, that said @SpaceInvaderOne was the one I got the idea from so others might be doing it too but not on the RC yet 8) 

    Link to comment

    I'm now on RC5. One thing I've noticed going down is that a second drive I've got mounted with Unassigned Devices seems to hang and be forced unmounted. Each time the array comes up it forces a parity check now. This started a few RC back but I figured it had to do with other things going on. It's formatted XFS and I see errors about the XFS drive not unmounting go by as it goes down and that drive is the only one I've got formatted XFS. Not clear to me what does it, I can say I don't stop VMs or containers before rebooting but this drive isn't used for that anyway <shrug>. Just mentioning it...

    Link to comment
    10 hours ago, BLKMGK said:

    I'm now on RC5. One thing I've noticed going down is that a second drive I've got mounted with Unassigned Devices seems to hang and be forced unmounted. Each time the array comes up it forces a parity check now. This started a few RC back but I figured it had to do with other things going on. It's formatted XFS and I see errors about the XFS drive not unmounting go by as it goes down and that drive is the only one I've got formatted XFS. Not clear to me what does it, I can say I don't stop VMs or containers before rebooting but this drive isn't used for that anyway <shrug>. Just mentioning it...

    I don't see what this has to do with this bug report.  Post in the UD forum and add diagnostics.  No one can help you based on this report of an issue with no supporting information.

    Link to comment

    I was mentioning it as an aside as it was something odd I had noticed and I'm not sure what's causing it, RC related or otherwise. You need not be so defensive.

    Link to comment

    This is resolved in rc6.  You will have to update UD before updating to rc6.

    Edited by dlandon
    • Like 1
    • Thanks 1
    Link to comment
    3 hours ago, dlandon said:

    This is resolved in rc6.  You will have to update UD before updating to rc6.

    Ah, the magical update to unRaid which doesn't actually exist yet ;) 

    Link to comment
    On 11/9/2019 at 8:29 PM, Squid said:

    Ah, the magical update to unRaid which doesn't actually exist yet ;) 

    You have to admit updates have been coming pretty quickly, thus far I've seen no huge showstoppers and am pretty excited about the new code! Lots of improvements and some of the speed issues of the past seem solved.

    Link to comment

    Just wanted to post and say THANK YOU!!!!!!

     

    Thanks to the efforts of the guys recompiling for the NVIDIA driver I was able to load up RC7 tonight and gve it a spin. The issues described above appears to be FIXED! My Go script no longer has to create a link to a cleartext password in order for me to boot my server - woohoo! I manually entered my password and the server started just fine - big Snoopy Dance! My thanks to the @limetech guys and to @dlandon for solvin this - much appreciated!!

    Link to comment

    Hi @BLKMGK , since I created my encrypted UD drive just like you did and I just stumbled upon this thread (still running on 6.7.2 but planning on updating to 6.8 soon, like maybe tomorrow)... does your last message mean I can update without a problem and my encrypted UD drive would mount when I start the array as usual and without problems?

    Link to comment



    Join the conversation

    You can post now and register later. If you have an account, sign in now to post with your account.
    Note: Your post will require moderator approval before it will be visible.

    Guest
    Add a comment...

    ×   Pasted as rich text.   Restore formatting

      Only 75 emoji are allowed.

    ×   Your link has been automatically embedded.   Display as a link instead

    ×   Your previous content has been restored.   Clear editor

    ×   You cannot paste images directly. Upload or insert images from URL.


  • Status Definitions

     

    Open = Under consideration.

     

    Solved = The issue has been resolved.

     

    Solved version = The issue has been resolved in the indicated release version.

     

    Closed = Feedback or opinion better posted on our forum for discussion. Also for reports we cannot reproduce or need more information. In this case just add a comment and we will review it again.

     

    Retest = Please retest in latest release.


    Priority Definitions

     

    Minor = Something not working correctly.

     

    Urgent = Server crash, data loss, or other showstopper.

     

    Annoyance = Doesn't affect functionality but should be fixed.

     

    Other = Announcement or other non-issue.