How to add L2ARC/SLOG/SPECIAL/DEDUP vdev to LUKS encrypted ZPOOL

[jxi7eoiigzns7oaf7]

I have a zpool of mirror vdevs which I created using the GUI and selecting 'zfs-encrypted' for filesystem type. I would like to add some nvme ssds as l2arc, slog, special, or dedup vdevs. I know this is not currently supported via the GUI, so I will have to use the CLI for this. I found JorgeB's excellent post here:

explaining how to add such vdevs to an existing zpool. However, since my devices are also LUKS encrypted I am not sure how best to go about this. I looked at

 

zpool history

 

to see how the original pool was created, and found this:

 

zpool create -f -o ashift=12 -o autotrim=on -O compression=on -O dnodesize=auto -O acltype=posixacl -O xattr=sa -O normalization=formD -m /mnt/tank tank mirror /dev/mapper/sdn1 /dev/mapper/sdo1 mirror /dev/mapper/sdp1 /dev/mapper/sdq1 mirror /dev/mapper/sdr1 /dev/mapper/sds1

 

So from that I gather 1) the devices are passed to zfs after the LUKS encryption is applied (makes sense), and 2) only one specific partition on each drive is passed to zfs, rather than the entire block device (also makes sense). In my case each drive only has one partition, but as a note I have also seen other posts mentioning that unraid only supports creating zpools this way using a single partition per drive at the moment, and that it HAS to be the first partition.

 

My question is: how do I accomplish this (i.e., a) encrypt the drives with LUKS, b) add them as vdevs to my zpool, and c) get unraid to import the pool automatically) without destroying my zpool or having any weird graphical issues like other people seem to be encountering while trying to modify their zpools via the CLI (e.g., drives not showing up after starting the array, missing drives from the GUI, being unable to format just the drives you're trying to add upon start of the array rather than being prompted to format every drive in your pool, etc.)?

Edited September 28, 2023 by jxi7eoiigzns7oaf7
typo

[JorgeB]

I never tested this with encryption, not sure it will be easy to make it work, because you'd need to encrypt the disks first, maybe if you first create a new separate pool with those devices and encrypt them, then and without wiping the devices just remove them from that pool and add them manually to the other one, but like mentioned it's something I never tried, alternatively wait for v6.13, it will have GUI support for hybrid pools.

[jxi7eoiigzns7oaf7]

Yeah I thought about either trying to encrypt them manually or with UD and then trying to pass the resulting LUKS volumes (partitions?) to 'zpool add' but I am not sure how Unraid handles opening encrypted volumes that you encrypt manually. I am going to see if there's a safe way I can test it with spare disks later when I get home. From what I can I see looking at the existing disks in my zpools and from watching the logs when I clear and format a drive with UD (both XFS - Encrypted and ZFS - Encrypted), all that's done is:

 

1) Wipe the partition table,

2) Create a new MBR partition table (in the case of SSDs),

3) cryptsetup luksFormat /dev/sdX

4) cryptsetup luksOpen /dev/sdX

 

which then gives you the expected /dev/mapper/sdX entry. However, I haven't yet found where in the config Unraid stores the list of devices it tries to mount/encrypted volumes it tries to open when you start the array. I am hoping if I can ensure these devices are added there that the pool import would be automatic after all the LUKS volumes are opened.

Edited September 28, 2023 by jxi7eoiigzns7oaf7

[JorgeB]

42 minutes ago, jxi7eoiigzns7oaf7 said:

I haven't yet found where in the config Unraid stores the list of devices it tries to mount/encrypted volumes it tries to open when you start the array.

AFAIK it will decrypt any assigned devices, so if you encrypt them now with UD or by assigning them to a different pool, they should be decrypted on array start once assigned to the other pool.

[JorgeB]

If I have some time I'll see if I can test this and add to the FAQ, but 6.13 should not be far off so not sure it worth it.

[JorgeB]

Was able to test and it works, but it's quite involved:

 

create new temp pool and assign all the devices you plan on adding
set fs to zfs encrypted, any profile
start array, format pool
stop array
unassign all devices from the temp pool and delete it
open new LUKS devices:
 

cryptsetup luksOpen /dev/sdX1 sdX1 --key-file=/root/keyfile

Replace X with correct letter, for NVMe devices it will be '/dev/nvmeXn1p1 nvmeXn1p1', do this for all new devices, if using a passphrase omit --key-file=/root/keyfile and enter the passphrase

Add the new vdevs as explained in the FAQ entry, just need to add mapper to every device, e.g.

zpool add tank -f special mirror /dev/mapper/sdf1 /dev/mapper/nvme0n1p1

stop array

Close LUKS for all new devices:

cryptsetup luksClose sdX1

do the pool import procedure as detailed in the FAQ

[jxi7eoiigzns7oaf7]

Awesome thank you very much for testing that out! I think I will try by just adding some L2ARC and hopefully nothing blows up if I make a mistake. That seems the easiest to undo if something goes awry. Do you know, if you've already started the array once and put in the disk encryption password in the process, and then stop the array, can you just do

 

cryptsetup luksOpen /dev/sdX1 sdX1

 

without specifying a keyfile or passphrase? Would it just use the one it has cached from you entering it to start the array? Presumably it's still cached since you can start the array again without re-entering the password. Is there some sort of 'crypto context' or something that would make this possible (and hopefully ensure the exact same settings are applied as when you use the GUI)?

 

I really just don't want to mess anything up by encrypting things incorrectly. I am not sure if Unraid uses a salt, for example, that wouldn't be automatically included even if I used the same password when doing the cryptsetup manually.

 

Thanks again for your help

Edited September 29, 2023 by jxi7eoiigzns7oaf7

[JorgeB]

2 minutes ago, jxi7eoiigzns7oaf7 said:

without specifying a keyfile or passphrase?

Keyfile will still be there with the array stopped, but you need to specify in the command, passphrase AFAIK you will always need to input manually, but I don't use encryption, so not very familiar with it.

[jxi7eoiigzns7oaf7]

So far I've added a single device L2ARC and a two device mirror as a SLOG using this method and it's gone well!