Help needed - UnRAID 6.6.6 server meltdown


jam

Recommended Posts

Hey all, I'm really hoping that someone here can help me out...

 

I have a server running UnRAID 6.6.6 (one of the DVB edition builds). Last night, I encountered a bit of a catastrophe. While using the web GUI, I clicked on the VM tab, and the entire machine hung - no services were accessible, I couldn't SSH in, and it wasn't even responding to pings. Connecting a display at this point showed nothing. This has happened a couple of times in the past, always when selecting the VM tab in the web interface (although this is rare; it's not a consistently reproducible problem), but as I couldn't do anything other than hit the reset button with no way I could think of to get any info out of the system, it just got added to my mental "I need to look into that at some point" to-do list.

 

This time, after rebooting, pretty much nothing seems to be working. The web interface isn't responding, and the array hasn't started (so obviously no Docker or VMs). I can SSH in (thankfully!) and the emhttpd process is running; I've been able to grab a diagnostics archive, but I'm at a bit of a loss as to what to do next. Any assistance would be appreciated!

holt-diagnostics-20190319-1930.zip

Link to comment

I'm no expert but it looks like something has happened to your array encryption? I assume your array is encrypted and this setting wasn't turned on by mistake or something of the like?

 

I don't use encryption in UnRAID so I'm trying to find some info for you, hopefully someone with experience with it can jump in and help.

Mar 19 19:08:03 Holt emhttpd: shcmd (41): udevadm settle
Mar 19 19:08:03 Holt emhttpd: Opening encrypted volumes...
Mar 19 19:08:03 Holt emhttpd: shcmd (43): /usr/sbin/cryptsetup luksOpen /dev/md1 md1 --key-file /root/keyfile
Mar 19 19:08:03 Holt kernel: device-mapper: ioctl: 4.39.0-ioctl (2018-04-03) initialised: [email protected]
Mar 19 19:08:03 Holt root: Failed to open key file.
Mar 19 19:08:03 Holt emhttpd: shcmd (43): exit status: 1
Mar 19 19:08:03 Holt emhttpd: shcmd (45): /usr/sbin/cryptsetup luksOpen /dev/md2 md2 --key-file /root/keyfile
Mar 19 19:08:03 Holt root: Failed to open key file.
Mar 19 19:08:03 Holt emhttpd: shcmd (45): exit status: 1
Mar 19 19:08:03 Holt emhttpd: shcmd (47): /usr/sbin/cryptsetup luksOpen /dev/md3 md3 --key-file /root/keyfile
Mar 19 19:08:03 Holt root: Failed to open key file.
Mar 19 19:08:03 Holt emhttpd: shcmd (47): exit status: 1
Mar 19 19:08:03 Holt emhttpd: shcmd (50): /usr/sbin/cryptsetup luksOpen /dev/md5 md5 --key-file /root/keyfile
Mar 19 19:08:03 Holt root: Failed to open key file.
Mar 19 19:08:03 Holt emhttpd: shcmd (50): exit status: 1
Mar 19 19:08:03 Holt emhttpd: shcmd (52): /usr/sbin/cryptsetup luksOpen /dev/md6 md6 --key-file /root/keyfile
Mar 19 19:08:03 Holt root: Failed to open key file.
Mar 19 19:08:03 Holt emhttpd: shcmd (52): exit status: 1
Mar 19 19:08:03 Holt emhttpd: shcmd (54): /usr/sbin/cryptsetup luksOpen /dev/md7 md7 --key-file /root/keyfile
Mar 19 19:08:03 Holt root: Failed to open key file.
Mar 19 19:08:03 Holt emhttpd: shcmd (54): exit status: 1
Mar 19 19:08:03 Holt emhttpd: shcmd (56): /usr/sbin/cryptsetup luksOpen /dev/nvme0n1p1 nvme0n1p1 --key-file /root/keyfile
Mar 19 19:08:03 Holt root: Failed to open key file.
Mar 19 19:08:03 Holt emhttpd: shcmd (56): exit status: 1
Mar 19 19:08:03 Holt emhttpd: shcmd (58): /usr/sbin/cryptsetup luksOpen /dev/nvme1n1p1 nvme1n1p1 --key-file /root/keyfile
Mar 19 19:08:03 Holt root: Failed to open key file.
Mar 19 19:08:04 Holt emhttpd: shcmd (58): exit status: 1
Mar 19 19:08:04 Holt emhttpd: Missing encryption key
Mar 19 19:08:04 Holt kernel: mdcmd (44): stop 

 

Edited by weirdcrap
Link to comment

Yes, the array is encrypted - the keyfile is located on another machine on my network. My server's boot drive contains the keys & scripts needed to fetch the keyfile & decrypt the array on startup, but you're right that this isn't happening. If this was the sole issue, though, I'd expect to still be able to access the web interface.

Edited by jam
Link to comment
13 minutes ago, jam said:

Yes, the array is encrypted - the keyfile is located on another machine on my network. My server's boot drive contains the keys & scripts needed to fetch the keyfile & decrypt the array on startup, but you're right that this isn't happening. If this was the sole issue, though, I'd expect to still be able to access the web interface.

I'm still going through your diagnostics (I am at work so I can only look intermittently) but I don't see anything else immediately wrong yet. I don't see any documented cases of failing to decrypt the disks causing the WebUI to fail to load either. I do see PHP-FPM and NGINX starting in the Syslog so I'll have to dig deeper into what else goes into getting the webUI started.

Edited by weirdcrap
Link to comment

Thanks, I appreciate the help. I don't need it up & running again urgently, but I'm wary of stumbling around by myself & making things worse. I've been able to boot the server when the keyfile has been unavailable in the past, and been able to access the web UI.

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.