Am trying to modify the fetch_key to use sftp over ssh with a password .ppk file

 

anyone have a good example?

 

Thanks

Myk

 

Edited May 29, 20188 yr by MyKroFt

Try:

#!/bin/bash

if [[ ! -e /root/keyfile ]]; then
  scp -i /path/to/ssh-private-key [email protected]:/path/to/keyfile /root/keyfile
fi

Just remember, the ssh-private-key should have permissions 600 as well as the directory its in, so you can't use the private key directly from the USB flash drive

Thanks for this fetch/delete script solution to keeping the keyfile off the server. Just a quick note from a problem I just ran into... 

 

If you add a disk to your array, obviously the keyfile isn't there anymore for unraid to encrypt the new drive. The error given in the webUI in 6.5.3 is a little ambiguous, after clearing the disk you click Format, then it refreshes and indicates the drive is encrypted with a green lock, but that it's unmountable because of a missing encryption key. Made me afraid I had encrypted it with the wrong key and would be spending many more hours clearing it again.

 

But once you put a keyfile in /root you can click Format again and it will format correctly, but just a heads up to remember you have to do this manually when adding a drive if you go out of your way to keep the keyfile off the running machine.

 
Thanks for the help

So I moved houses and me server was powered off. Once powered in it did not have internet access as I wanted to configure the network before putting my server online. I discovered that even with out internet access my array auto mounted
I was under the impression if there was no internet access to go out to the remote location ( using ftp) to get the key it would not start.

Is this true? Or did I miss something?

Sent from my BND-L34 using Tapatalk

Array will start when autostart is set to enabled regardless of internet access.

When the encryption key is missing it will not mount those devices which have encryption.

 

Ok that makes sense, thanks

Sent from my BND-L34 using Tapatalk

On 4/2/2018 at 9:01 AM, bonienl said:

A small variation if you want the key to be not locally present on the system when operational, the key is only needed during startup of the array.

 

In the go file the following is included before starting emhttp.

# auto unlock array
mkdir -p /usr/local/emhttp/webGui/event/starting
mkdir -p /usr/local/emhttp/webGui/event/started
mkdir -p /usr/local/emhttp/webGui/event/stopped
cp -f /boot/custom/bin/fetch_key /usr/local/emhttp/webGui/event/starting
cp -f /boot/custom/bin/delete_key /usr/local/emhttp/webGui/event/started
cp -f /boot/custom/bin/fetch_key /usr/local/emhttp/webGui/event/stopped

# start webGUI
/usr/local/sbin/emhttp &

The above makes use of the built-in event system of unRAID. These events are created:

starting : this event is called before the array is started and is used to fetch the key from a remote source

started :  this event is called after the array is fully operational and is used to delete the key locally.

stopped : this event is called after the array is stopped and is used to fetch the key again from a remote source

 

The script "fetch_key"  can be any method to obtain the key remotely, e.g. using a mount method or a FTP (wget) method as explained in the video of @gridrunner

The script "delete_key" is a simple file to delete the key locally.

 

fetch_key

#!/bin/bash

if [[ ! -e /root/keyfile ]]; then
  mkdir -p /unlock
  mount -t cifs -o user=name,password=password,iocharset=utf8 //192.168.1.99/index /unlock
  cp -f /unlock/somefile.png /root/keyfile
  umount /unlock
  rm -r /unlock
fi

delete_key

#!/bin/bash

rm -f /root/keyfile

You can start and stop the array as usual, and the key will be automatically fetched each time, provided that the remote service is up and running.

 

The files "fetch_key" and "delete_key" need to be stored on your flash device. I've created the folder /custom/bin to hold my custom scripts, but one is free to choose their own source folder, please update the lines in the go file accordingly.

I need some assistance getting this to work.  I add the two scripts to the flash.  the fetch_key doesn't load the key file from my windows 10 computer.

 

I checked the firewalls and that's not an issue.  What else could it be?

 

What log should I be referencing to see what happens during the boot?

Can I use a SMB share to access the keyphase?

On 9/6/2018 at 10:52 PM, sgt_spike said:

I need some assistance getting this to work.

Open a terminal session and copy/paste each line (skip the 'if' condition) to see the replies (errors)

6 hours ago, sgt_spike said:

Can I use a SMB share to access the keyphase?

Yes, if you mean an SMB share on a different system.

The code example in the post above yours is exactly doing that.

4 minutes ago, bonienl said:

Open a terminal session and copy/paste each line (skip the 'if' condition) to see the replies (errors)

Thanks for the guidance.  trying it now

 

55 minutes ago, bonienl said:

Open a terminal session and copy/paste each line (skip the 'if' condition) to see the replies (errors)

Didn't see any errors.  I modified the fetch key to the following

if [[ ! -e /root/keyfile ]]; then
  mkdir -p /unlock
  mount -t cifs -o rw,nounix,iocharset=utf8,_netdev,file_mode=0777,dir_mode=0777,,vers=3.0,username=myuser,password=mypassword '//MEDIASERVER/index' '/mnt/disks/MEDIASERVER_index'
  cp -f /mnt/disks/MEDIASERVER_index/unlock.png /root/keyfile
#  umount /unlock
  rm -r /unlock
fi

restarted the server and got the following

Sep 23 17:09:00 Tower emhttpd: shcmd (32): echo 128 > /sys/block/sdb/queue/nr_requests
Sep 23 17:09:00 Tower emhttpd: shcmd (33): /usr/local/sbin/set_ncq sdd 1
Sep 23 17:09:00 Tower root: set_ncq: setting sdd queue_depth to 1
Sep 23 17:09:00 Tower emhttpd: shcmd (34): echo 128 > /sys/block/sdd/queue/nr_requests
Sep 23 17:09:00 Tower kernel: mdcmd (40): start STOPPED
Sep 23 17:09:00 Tower kernel: unraid: allocating 31100K for 1280 stripes (6 disks)
Sep 23 17:09:00 Tower kernel: md1: running, size: 3907018532 blocks
Sep 23 17:09:00 Tower kernel: md2: running, size: 3907018532 blocks
Sep 23 17:09:00 Tower kernel: md3: running, size: 2930266532 blocks
Sep 23 17:09:00 Tower kernel: md4: running, size: 1953514552 blocks
Sep 23 17:09:01 Tower emhttpd: shcmd (35): udevadm settle
Sep 23 17:09:01 Tower emhttpd: Opening encrypted volumes...
Sep 23 17:09:01 Tower emhttpd: shcmd (37): /usr/sbin/cryptsetup luksOpen /dev/md1 md1 --key-file /root/keyfile
Sep 23 17:09:01 Tower root: Failed to open key file.
Sep 23 17:09:01 Tower kernel: device-mapper: ioctl: 4.39.0-ioctl (2018-04-03) initialised: [email protected]
Sep 23 17:09:01 Tower emhttpd: shcmd (37): exit status: 1
Sep 23 17:09:01 Tower emhttpd: shcmd (39): /usr/sbin/cryptsetup luksOpen /dev/md2 md2 --key-file /root/keyfile
Sep 23 17:09:01 Tower root: Failed to open key file.
Sep 23 17:09:01 Tower emhttpd: shcmd (39): exit status: 1
Sep 23 17:09:01 Tower emhttpd: shcmd (41): /usr/sbin/cryptsetup luksOpen /dev/md3 md3 --key-file /root/keyfile
Sep 23 17:09:01 Tower root: Failed to open key file.
Sep 23 17:09:01 Tower emhttpd: shcmd (41): exit status: 1
Sep 23 17:09:01 Tower emhttpd: shcmd (43): /usr/sbin/cryptsetup luksOpen /dev/md4 md4 --key-file /root/keyfile
Sep 23 17:09:01 Tower root: Failed to open key file.
Sep 23 17:09:01 Tower avahi-daemon[11634]: Server startup complete. Host name is Tower.local. Local service cookie is 2421470760.
Sep 23 17:09:01 Tower emhttpd: shcmd (43): exit status: 1
Sep 23 17:09:01 Tower emhttpd: shcmd (45): /usr/sbin/cryptsetup luksOpen /dev/sdf1 sdf1 --key-file /root/keyfile
Sep 23 17:09:01 Tower root: Failed to open key file.
Sep 23 17:09:01 Tower emhttpd: shcmd (45): exit status: 1
Sep 23 17:09:01 Tower emhttpd: Missing encryption key
Sep 23 17:09:01 Tower kernel: mdcmd (41): stop 
Sep 23 17:09:01 Tower kernel: md1: stopping
Sep 23 17:09:01 Tower kernel: md2: stopping
Sep 23 17:09:01 Tower kernel: md3: stopping
Sep 23 17:09:01 Tower kernel: md4: stopping
Sep 23 17:09:02 Tower avahi-daemon[11634]: Service "Tower" (/services/ssh.service) successfully established.
Sep 23 17:09:02 Tower avahi-daemon[11634]: Service "Tower" (/services/smb.service) successfully established.
Sep 23 17:09:02 Tower avahi-daemon[11634]: Service "Tower" (/services/sftp-ssh.service) successfully established.

You have a chicken and egg problem...

You want to mount /mnt/disks/... but this location is not existing yet because the key is missing.

I used the temporary folder /unlock which is created in RAM. This is possbile at any time, you need to do something similar.

On 12/9/2017 at 9:13 AM, bonienl said:

unRAID 6.4 introduces encrypted volumes and allows individual disks to be protected by encryption.

The passphrase or keyfile which is used to unlock encryption is not permanently stored on the system itself, this means that upon a system start one has to re-enter the passphrase or re-select the keyfile. Until then the array can not start and the GUI will report a "missing key".

 

But what if we want to auto-start the array and not keep a local copy on the system itself? It kind of defeats the protection scheme to store a local copy, i.e. when your system get stolen, you don't want to include the keys to allow access.

 

The solution I have come up is to download the keyfile during startup from a different system in my local network which is in a different part of the house. In my case I am using an arbitrary .png file, but you could also store a passphrase and have this copied over. In my go file I added the following lines to download the keyfile from a remote SMB share before starting emhttp.

# auto unlock array
mkdir -p /unlock
mount -t cifs -o user=name,password=password,iocharset=utf8 //192.168.1.123/index /unlock
cp -f /unlock/#/some.png /root/keyfile
umount /unlock
rm -r /unlock

Of course the remote share must be accessible when the system starts and surely this approach can be cracked, but it is definitely safer than just keeping a local copy of the keyfile on the unRAID server itself.

 

Remember that once the array is started, the local keyfile may be deleted if you really don't want to keep the key while the system is operational.

 

Perhaps my idea is useful for others wanting to do something similar

 

THANK YOU!!!

 

I am adapting this to simply:

# auto unlock array
cp -f /boot/config/keyfile /root/keyfile

I copied the mounted keyfile under /root out to /boot/config/keyfile and placed this as first thing in go script.

Hope it works, rebooting now. My reasons for not having to use some other devices are simple, I still want encryption and automatic mounting without wrecking my docker containers on my encrypted and unassigned SSD and if I ever want, can just get a USB drive out and plug it into unraid as "Encryption key" or something if I need that level of safety.

 

UPDATE: It worked!

Edited January 5, 20197 yr by Trunkton

So I've done a semi-equivalent of what you guys are doing here except I'm using scp to pull my key down from an AWS micro instance (might switch this to a Pi Zero on premise though - thoughts on drawbacks/benefits here?) that is always on.  I like the examples where I can delete the key after the disks are mounted which is why I stumbled onto this thread.

 

One idea I had was to somehow enhance the scp/ftp/whatever protocol process to somehow fire off a process on the remote machine that holds the key to send maybe an SMS text to my phone where I respond with a PIN code to allow the key transmission to take place.  Are there any inherent flaws with pursuing this?  One concern I have is there will obviously be a lag between when the ssh command between Unraid go script and the key and be transmitted back and then it's a race condition?  Is there a shell command on the Unraid side that I can call again to mount the drives once the file has been received?

Hi everybody.

Just to be sure: if I encrypt only my data disks but not cache, will all docker containers (set to cache only) autostart with server boot or do I need to enter passphrase?

I'm asking this because I use an open VPN server, and if I'd remotely reboot the server I couldn't access web gui and thus couldnt start the array at all and think about your autostart strategies.

thanks so much

When encryption is used then a passphrase needs to be entered to start the array.

Both data disks and cache disks are part of the array.

The open vm plugin will let you access the server when its stopped

Sent via telekinesis

4 hours ago, melagodo said:

I use an open VPN server, and if I'd remotely reboot the server I couldn't access web gui

It would probably be a good idea to set up a backup contingency, if your router has a VPN server set that up as a backup, or maybe another machine on the same lan with teamviewer.

I thought I'd share how you can enhanced the go file by reducing the six lines to a single command and it's not by using another script.  You can create a tar ball that contains the fetch_key and delete_key scripts. The go file calls the tar command. The tar ball files are extracted and event directories are created.

 

You MUST have a fully functioning auto-start that unlocks using the event directories. This works with FTP or SMB fetch_key scripts.

 

If you have changed the script names (fetch_key, delete_key) or changed the path where you store the scripts (/boot/custom/bin/), you will need to use your alternative names in the following procedure.

 

1) Create a tar ball call "events" from the existing files in the event directories.

 

At the terminal prompt enter the following:

tar -czf /boot/custom/bin/events -C /usr/local/ emhttp/webGui/event/starting/fetch_key emhttp/webGui/event/started/delete_key emhttp/webGui/event/stopped/fetch_key

2) Update the go file. Comment out the existing lines in order to test.

 

From:-

# auto unlock array
mkdir -p /usr/local/emhttp/webGui/event/starting
mkdir -p /usr/local/emhttp/webGui/event/started
mkdir -p /usr/local/emhttp/webGui/event/stopped
cp -f /boot/custom/bin/fetch_key /usr/local/emhttp/webGui/event/starting
cp -f /boot/custom/bin/delete_key /usr/local/emhttp/webGui/event/started
cp -f /boot/custom/bin/fetch_key /usr/local/emhttp/webGui/event/stopped

To:

# auto unlock array
# mkdir -p /usr/local/emhttp/webGui/event/starting
# mkdir -p /usr/local/emhttp/webGui/event/started
# mkdir -p /usr/local/emhttp/webGui/event/stopped
# cp -f /boot/custom/bin/fetch_key /usr/local/emhttp/webGui/event/starting
# cp -f /boot/custom/bin/delete_key /usr/local/emhttp/webGui/event/started
# cp -f /boot/custom/bin/fetch_key /usr/local/emhttp/webGui/event/stopped
tar -xzf /boot/custom/bin/events -C /usr/local/

3) Once you're confident that everything works, rebooting IS necessary. You can clean up by deleting the event scripts (fetch_key, delete_key). Your files are now stored in the "event" tar ball.  And, updating the go file by removing the commented lines and any references to "unlock".

# auto start array
tar -xzf /boot/custom/bin/events -C /usr/local/

I hope some of you may find this interesting.

Edited November 26, 20205 yr by beckp
error in the created tar command line

This seem to be broken in the latest version (6.8.0-rc3), looks like the startup/shutdown scripts are not executing. Has anyone got this to work with the latest RC version?

32 minutes ago, Nalith said:

This seem to be broken in the latest version (6.8.0-rc3), looks like the startup/shutdown scripts are not executing. Has anyone got this to work with the latest RC version?

Have you checked that they have the ‘execute’ bit set?     One of the changes in the 6.8 series is to limit the attributes of files on the flash drive.  This may mean you need to explicitly set the ‘x’ bit after copying the files off the flash drive.

On 10/22/2019 at 5:31 PM, itimpi said:

Have you checked that they have the ‘execute’ bit set?     One of the changes in the 6.8 series is to limit the attributes of files on the flash drive.  This may mean you need to explicitly set the ‘x’ bit after copying the files off the flash drive.

Thanks for the prompt response. I also thought that is the issue however for the life of me it seems the chmod changes are not being applied: 

 

 

Is there something I am missing?

23 minutes ago, Nalith said:

Thanks for the prompt response. I also thought that is the issue however for the life of me it seems the chmod changes are not being applied: 

 

 

Is there something I am missing?

 

The tightening of permissions on the boot drive for security reasons means it is no longer possible to have files stored on that drive with execute permissions set.   The files need to be copied off the drive to somewhere else before you can set execute permissions.

 

All the methods above are using servers on the lan or remotely, which I can see the use case for, but could Unraid also use a similar method as Bitlockered Microsoft Windows, using an inserted USB flash drive which has the keyfile on it.