Backup and restore of LUKS headers on ARRAY data drives

Opawesome · January 20, 2023

A few month ago I tried writing a script in order to backup and restore LUKS headers on encrypted drives. Some interesting issues were raised by @Kilrah here, highlighting in particular the fact that the first 32kB of every drive is "reserved" by Unraid on array data drives, which could cause issues.

Has anyone input to provide on the following ?

should the "cryptsetup luksHeaderBackup" command be run on "/dev/sdX" (the raw drive) or on "/dev/mdX" (the mounted volume) on array data drives?
should the "cryptsetup luksHeaderBackup" command be run with the array started or the array stopped, and how would that affect parity?
should the "cryptsetup luksHeaderRestore" command be run on "/dev/sdX" (the raw drive) or on "/dev/mdX" (the mounted volume) on array data drives?
should the "cryptsetup luksHeaderRestore" command be run with the array started or the array stopped, and how would that affect parity?

I thank you very much in advance.

As I said earlier, I would love to see @limetech implement backup/restore of LUKS headers directly into Unraid, as I feel I really should not play with theses scary headers.

With regards,

OP

Edited January 20, 2023 by Opawesome

Opawesome · January 20, 2023

After some testing, it seems that "/dev/sdX" should maybe be avoided and that "/dev/sdX1" (the partition) shall be preferred.

It would be awesome if someone knowledgeable could confirm.

JorgeB · January 20, 2023

Using sdX1 would be the same as using mdX, except it won't maintain parity, and if the LUKS headers are on the partition they should be included in parity, but I've never used encryption, so not sure what would be the best way, maybe you could ask the user to start the array in militance mode when a restore is needed and that way you could use mdX.

Opawesome · January 20, 2023

4 minutes ago, JorgeB said:

Using sdX1 would be the same as using mdX, except it won't maintain parity, and if the LUKS headers are on the partition they should be included in parity, but I've never used encryption, so not sure what would be the best way, maybe you could ask the user to start the array in militance mode when a restore is needed and that way you could use mdX.

Many Thanks. This is useful. I will have to check whether the array can be started (even in maintenance mode) if the LUKS header of one array data drive is missing, because I think I remember that an error is issues by Unraid about "missing encryption key".

JorgeB · January 20, 2023

6 minutes ago, Opawesome said:

because I think I remember that an error is issues by Unraid about "missing encryption key".

The array should still start and display that error, at least pretty sure it does.

Opawesome · January 20, 2023

OK. I will try and revert back. Many thanks.

Opawesome · January 21, 2023

17 hours ago, JorgeB said:

The array should still start and display that error, at least pretty sure it does.

I checked and the array does not start (even in maintenance mode) if one data drive doesn't have the proper LUKS key is its header:

Jan 21 11:24:29 BEETHOVEN kernel: mdcmd (117): set md_num_stripes 1280
Jan 21 11:24:29 BEETHOVEN kernel: mdcmd (118): set md_queue_limit 80
Jan 21 11:24:29 BEETHOVEN kernel: mdcmd (119): set md_sync_limit 5
Jan 21 11:24:29 BEETHOVEN kernel: mdcmd (120): set md_write_method
Jan 21 11:24:29 BEETHOVEN kernel: mdcmd (121): start STOPPED
Jan 21 11:24:29 BEETHOVEN kernel: unraid: allocating 20870K for 1280 stripes (4 disks)
Jan 21 11:24:29 BEETHOVEN kernel: md1: running, size: 2097120 blocks
Jan 21 11:24:29 BEETHOVEN kernel: md2: running, size: 2097120 blocks
Jan 21 11:24:29 BEETHOVEN emhttpd: shcmd (5231): udevadm settle
Jan 21 11:24:29 BEETHOVEN emhttpd: Opening encrypted volumes...
Jan 21 11:24:29 BEETHOVEN emhttpd: shcmd (5234): /usr/sbin/cryptsetup luksOpen /dev/md1 md1 --allow-discards
Jan 21 11:24:29 BEETHOVEN emhttpd: shcmd (5237): /usr/sbin/cryptsetup luksOpen /dev/md2 md2 --allow-discards
Jan 21 11:24:31 BEETHOVEN emhttpd: shcmd (5240): /usr/sbin/cryptsetup luksOpen /dev/sdc1 sdc1 --allow-discards
Jan 21 11:24:33 BEETHOVEN emhttpd: Missing encryption key
Jan 21 11:24:33 BEETHOVEN emhttpd: shcmd (5241): /usr/sbin/cryptsetup luksClose md2
Jan 21 11:24:33 BEETHOVEN emhttpd: shcmd (5242): /usr/sbin/cryptsetup luksClose sdc1
Jan 21 11:24:33 BEETHOVEN kernel: mdcmd (122): stop
Jan 21 11:24:33 BEETHOVEN kernel: md1: stopping
Jan 21 11:24:33 BEETHOVEN kernel: md2: stopping

So the operations should be performed on /dev/sdX1 I guess.

Edited January 21, 2023 by Opawesome

JorgeB · January 21, 2023

13 minutes ago, Opawesome said:
Jan 21 11:24:33 BEETHOVEN emhttpd: Missing encryption key

Did you enter the encryption key?

Opawesome · January 21, 2023

2 minutes ago, JorgeB said:

Did you enter the encryption key?

Yes.

But in any case, I think that the encryption key is in the LUKS header, and is not the passphrase, which is only a way to decipher the actual encryption key in the LUKS header.

Also, I think the encryption keys in the LUKS header are different on each drive.

Edited January 21, 2023 by Opawesome

JorgeB · January 21, 2023

If I set the devices to encrypted, and they are not, so there's no LUKS heathers, I can still start the array after entering a key:

Kilrah · January 21, 2023

21 minutes ago, Opawesome said:

I checked and the array does not start (even in maintenance mode) if one data drive doesn't have the proper LUKS key is its header:

That makes no sense since when you add a new drive it has no header until you format it, and obviously you need to start the array for that...

Or is it a problem only if the header is there but not the key?

Edited January 21, 2023 by Kilrah

Opawesome · January 21, 2023

I did further testing:

If I run "cryptsetup erase" on an encrypted drive to clear the LUKS encryption key on that drive (but maintaining the LUKS header) then the array won't start, even if I tick "maintenance mode".

If I run "cryptsetup erase" on an encrypted drive to clear the LUKS encryption key on that drive and then also "wipefs -a /dev/sdX1" to erase the LUKS remaining header (as instructed here), then the array will start, as on your system.

Kilrah · January 21, 2023

18 minutes ago, Opawesome said:

If I run "cryptsetup erase" on an encrypted drive to clear the LUKS encryption key on that drive (but maintaining the LUKS header) then the array won't start, even if I tick "maintenance mode".

I've just tried this and it worked here.

array started in normal mode
run backup script
cryptsetup erase /dev/md2
reboot so the passphrase isn't cached
enter passphrase, start array in normal mode - no problem, just obviously the disk is unmountable ("Unmountable: Volume not encrypted")
cryptsetup luksHeaderRestore /dev/md2 --header-backup-file /boot/luks/luks-headers-backup-20230121115834/0QEMU_QEMU_HARDDISK_drive-scsi4.bin
stop array
start array in normal mode
disk mounts fine
parity check returns no error

Edited January 21, 2023 by Kilrah

Opawesome · January 21, 2023

3 minutes ago, Kilrah said:

I've just tried this and it worked here.

array started in normal mode

run backup script

cryptsetup erase /dev/md2

reboot so the passphrase isn't cached

enter passphrase, start array in normal mode - no problem, just obviously the disk is unmountable ("Unmountable: Volume not encrypted")

cryptsetup luksHeaderRestore /dev/md2 --header-backup-file /boot/luks/luks-headers-backup-20230121115834/0QEMU_QEMU_HARDDISK_drive-scsi4.bin

stop array

start array in normal mode

disk mounts fine

parity check returns no error

On my former test I used "cryptsetup erase /dev/sdx1".

So I did like you:

array started in normal mode
run backup script
cryptsetup erase /dev/md1 (variation in my testing)
stop array
reboot (new step in my testing)
> the array fails to start, as with "cryptsetup erase /dev/sdx1"

Our results are clearly different..

FYI I use Unraid 6.9.2 on my testing VM.

Kilrah · January 21, 2023

2nd try with

dd if=/dev/zero of=/dev/md2 bs=1 count=16777216 to entirely wipe the 16MiB LUKS header after the cryptsetup erase

Same, no issue starting array and restore worked, parity maintained all the way.

7 minutes ago, Opawesome said:

FYI I use Unraid 6.9.2 on my testing VM.

Current 6.11.5 here.

Also I'm using a passphrase, not a keyfile.

Edited January 21, 2023 by Kilrah

Opawesome · January 21, 2023

1 hour ago, Kilrah said:

Also I'm using a passphrase, not a keyfile.

Likewise.

Opawesome · January 21, 2023

2 hours ago, Kilrah said:

2nd try with

dd if=/dev/zero of=/dev/md2 bs=1 count=16777216 to entirely wipe the 16MiB LUKS header after the cryptsetup erase

Same, no issue starting array and restore worked, parity maintained all the way.

I tried on my system and had parity errors...:

root@BEETHOVEN:/boot/luks# cryptsetup erase /dev/md1

WARNING!
========
This operation will erase all keyslots on device /dev/md1.
Device will become unusable after this operation.

Are you sure? (Type 'yes' in capital letters): YES
root@BEETHOVEN:/boot/luks# wipefs -a /dev/md1
wipefs: error: /dev/md1: probing initialization failed: Device or resource busy
root@BEETHOVEN:/boot/luks# dd if=/dev/zero of=/dev/md1 bs=1 count=16777216
16777216+0 records in
16777216+0 records out
16777216 bytes (17 MB, 16 MiB) copied, 17.0588 s, 983 kB/s
root@BEETHOVEN:/boot/luks# cryptsetup luksHeaderRestore /dev/md1 --header-backup-file "QEMU_HARDDISK_QM00011.bin"

WARNING!
========
Device /dev/md1 does not contain LUKS2 header. Replacing header can destroy data on that device.

Are you sure? (Type 'yes' in capital letters): YES
root@BEETHOVEN:/boot/luks#

Last check completed on Sat 21 Jan 2023 03:14:40 PM CET (today)
Finding 63 errors Duration: 13 seconds. Average speed: 247.8 MB/sec

Edited January 21, 2023 by Opawesome

Opawesome · January 21, 2023

7 hours ago, Kilrah said:

2nd try with

dd if=/dev/zero of=/dev/md2 bs=1 count=16777216 to entirely wipe the 16MiB LUKS header after the cryptsetup erase

Same, no issue starting array and restore worked, parity maintained all the way.

Current 6.11.5 here.

Also I'm using a passphrase, not a keyfile.

After updating to Unraid 6.11.5, I finally get the same results as you:

I was able to start the array in maintenance mode after deleting only the LUKS keys.

I was able to wipe the LUKS header completely with "wipefs -a /dev/md1" (but I had to restart the array in maintenance mode).

I was able to restore the LUKS header on /dev/mdX (i.e. with the array started in maintenance mode)

I did not get any error after a performing a parity check after all that.

I opened a new thread here to get additional help regarding how to obtain the SERIAL_ID of the disk hosting a certain dev/mdX managed partition.

Backup and restore of LUKS headers on ARRAY data drives

Recommended Posts

Opawesome

Link to comment

Opawesome

Link to comment

JorgeB

Link to comment

Opawesome

Link to comment

JorgeB

Link to comment

Opawesome

Link to comment

Opawesome

Link to comment

JorgeB

Link to comment

Opawesome

Link to comment

JorgeB

Link to comment

Kilrah

Link to comment

Opawesome

Link to comment

Kilrah

Link to comment

Opawesome

Link to comment

Kilrah

Link to comment

Opawesome

Link to comment

Opawesome

Link to comment

Opawesome

Link to comment

Join the conversation