[Script] Backup LUKS headers of encrypted devices


Recommended Posts

Hi all,

 

In the .zip archive file attached to this post is a script I have written in order to backup LUKS headers of encrypted devices attached to Unraid.

 

1. WHY?


Backing up LUKS headers is a paramount precautionary measure given the fact that data corruption in LUKS header may lead to losing all the data present on the device. Please note that in this scenario, knowing the passphrase (or having a copy of the key file used to store the passphrase) is irrelevant and will not help you recover the data. This is because the passphrase is not used in deriving the encryption master key, but only used in decrypting the encryption master key, the latter being randomly generated on each device upon initial creation of the LUKS header. 

 

2. HOW?

 

To backup your LUKS headers, just run the attached script as root. The script is not interactive, therefore you can use it with the wonderful User.Script plugin by @Squid  (see: here).

 

3. WHEN?

 

LUKS headers only need to be backed up upon a change in one of the headers. A change should occur upon events such as:

 

  • any drive is formatted (or reformatted) as encrypted (this includes the addition of a new encrypted drive)
  • a passphrase (or the key file used to store the passphrase) is added to, changed in or removed from any of the 8 different key slots provided by LUKS (the Unraid GUI allows you to create only one passphrase for the entire array, but the underlying kernel module does allow 8 different passphrases for each encrypted device)

 

Therefore, there is no need to schedule the backup to any particular period. A scheduled backup should even be avoided if you elect to copy the backed up LUKS headers to the Unraid flash drive, as it is considered good practice to keep the read/write operations of the flash drive to a minimum.

 

4. HOW DO I RESTORE LUKS HEADERS FROM A PREVIOUS BACKUP?

 

As of now, the safest way I have found to restore LUKS headers from a previous backup is to run the command below manually for each drive that needs to have its LUKS header restored (don't forget to replace the <path to relevant LUKS header backup> and <device / device partition> with the correct information for your use case):

 

cryptsetup luksHeaderRestore --header-backup-file <path to relevant LUKS header backup> <device / device partition>

 

IMPORTANT: if the <device / device partition> is assigned to an array protected by parity, ensure that you are pointing to the managed partition device, i.e.: "/dev/mdXp1" (start the array in maintenance mode first) and not to the actual "dev/sdY1" partition device. Indeed, writes to /dev/sdY1 will bypass parity but writes to /dev/mdXp1 do not.

 

5. WORK IN PROGRESS

 

I am trying to write a command that would restore all LUKS header at once. However, I need to find a way to reconcile (in CLI) each "/dev/mdXp1" device (i.e.: the device that is created after starting the array and which needs to be written on for the restore operation in order to preserve parity) to the corresponding "/dev/sdY" device (i.e.: the actual device which contains the "ID_SERIAL" used to name the LUKS header backup .bin file for that disk).

 

I asked the question here but never got an answer.

 

The command would be something like (note that you should not run it as is because it is not working in its current state):

 

# DO NOT USE / DO NOT USE / DO NOT USE
#cd "/directory/where/the/backed/up/LUKS/headers/are/stored" && for i in {/dev/sd*,/dev/nvme*}; do if cryptsetup luksDump $i &>/dev/null; then cryptsetup luksHeaderRestore <WIP: '$i' cannot be used here because it would bypass parity> --header-backup-file "`udevadm info --query=all --name=$i | sed -n 's/.*ID_SERIAL=//p'`.bin"; fi; done
# DO NOT USE / DO NOT USE / DO NOT USE

 

5. ANYTHING ELSE?

 

Feel free to report any issues and make any suggestions.

 

This script is based on exchanges from the following threads, which you can refer to: 

as well as on the official LUKS page :

 

Best,

OP

 

 

unraid_backup-luks-headers-v0.2a.zip

Edited by Opawesome
Added new section 3: "WHEN?"; Added clarification that the script is in the attached .zip archive file; Updated script tp fix error when disk's SERIAL_ID contains space characters; Added noew section 4 HOW DO I RESTORE LUKS HEADERS FROM A PREVIOUS BACKUP
  • Like 5
  • Thanks 1
Link to comment
  • Opawesome changed the title to [Script] Backup LUKS headers of encrypted devices

Just a note of caution. Only write to the USB when absolutely necessary. Unraid takes great pains to limit the data written to the boot USB, and needlessly scheduling this script to run periodically could be counterproductive.

I recommend manually running this when you make a change, don't schedule it to make it convenient.

  • Like 1
Link to comment
2 hours ago, jonathanm said:

Just a note of caution. Only write to the USB when absolutely necessary. Unraid takes great pains to limit the data written to the boot USB, and needlessly scheduling this script to run periodically could be counterproductive.

I recommend manually running this when you make a change, don't schedule it to make it convenient.

 

Thank you @jonathanm. Very relevant comment. I added a new section to the initial post: "3. WHEN?", in order to address this issue.

 

Best,

OP

  • Like 1
Link to comment
  • 8 months later...
  • 4 weeks later...

Hi @Opawesome,

 

I've setup my first unraid server recently, so a total newbie at unraid. I've got my array/cache set to XFS - encrypted, with the hopes of should a drive ever fail I can RMA it without having to worry about data that might be on the drives.

 

When I run the script however, I get the following (I've hashed out the last few digits of S/N, and I get this message for all drives):

"Device ST4000VN008-2DR166_########.bin does not exist or access denied.
Header backup file does not contain compatible LUKS header."

 

Could you perhaps expand on the "How" section a bit more? As I'm missing something, but not sure what.

Have you managed to test the Restore functionality by any chance?

 

Regards,

Grimlious

Link to comment
7 hours ago, Grimlious said:

When I run the script however, I get the following (I've hashed out the last few digits of S/N, and I get this message for all drives):

"Device ST4000VN008-2DR166_########.bin does not exist or access denied.
Header backup file does not contain compatible LUKS header."

 

Hi @Grimlious,

 

Reading that error, it looks like you are trying to run the "restore" command i mentionned in my original post and not the script inside the .zip archive attached to it.

 

Can you confirm that you downloaded the attached .zip file and run the script contained therein ?

 

With regards,

OP

Edited by Opawesome
  • Haha 1
Link to comment
  • 5 months later...

Nice script, thank you!  but i think that it needs some little rework.

It's not possible to backup the luks headers of several encrypted nvme's.

 

I have 3 Samsung Nvme and its only generating one file named Samsung in the backup location for them. Also i received the following error during the process ->

 

Script location: /tmp/user.scripts/tmpScripts/unraid_backup-luks-headers/script
Note that closing this window will abort the execution of this script
This script will backup the LUKS headers of all LUKS encrypted devices which are present on this host (Sokrates) to the following directory: /mnt/disks/2116E598E8E4/luksheaders/.
This script will also copy the backed up LUKS headers to the Unraid flash drive, under the following directory: /boot/config/luks
Now starting backup of LUKS headers...
Requested header backup file Samsung already exists.
Requested header backup file Samsung already exists.
Backup of LUKS headers failed. Now exiting...
/tmp/user.scripts/tmpScripts/unraid_backup-luks-headers/script: line 80: return: can only `return' from a function or sourced script
Now setting proper owner and permissions for backed up LUKS headers...
Now starting copy of backed up LUKS headers to the Unraid flash drive...
Copy of backed up LUKS headers to the Unraid flash drive completed successfully.
Now exiting...

 

 

This worked for me, backed up and copyed this for my nvmes manually, with this example:

cryptsetup luksHeaderBackup /dev/nvme1n1p1 --header-backup-file=/backuplocation/file.bin

 

Edited by T-Birth
  • Thanks 1
Link to comment
On 8/14/2022 at 6:57 PM, T-Birth said:

Nice script, thank you!  but i think that it needs some little rework.

It's not possible to backup the luks headers of several encrypted nvme's.

 

I have 3 Samsung Nvme and its only generating one file named Samsung in the backup location for them. Also i received the following error during the process [...]

 

Thank you @T-Birth for reporting the error. Indeed, I also have this error since I upgraded Unraid to a version above 6.8.3.

 

I have updated the script and backup command to fix it.

 

It turned out that I needed to add some double quotes because some disks' SERIAL_ID now have space characters in them.

 

I successfully tested the backup script (but not the restore command) on Unraid 6.10.3.

Link to comment

I don't know if anyone ever tried the restore method until now, but I just did and here are some notes, as far as I can see the described method can't work at least on array drives:

 

- Backed up with the script, with array running

- Using info from https://wiki.archlinux.org/title/Dm-crypt/Drive_preparation#Wipe_LUKS_header I erased all keys on an array drive and the header

- Tried restoring with the restore command, both with array stopped and started

 

This only asked me to confirm overwriting of existing headers on the other drives, not about the one that had it wiped. Seems the "if cryptsetup luksDump $i &>/dev/null; then" purposefully would not include any disk that does not already have a valid header, seems the opposite of what is wanted

 

Manually running

cryptsetup luksHeaderRestore /dev/sdd --header-backup-file /boot/luks/luks-headers-backup-20230120132359/0QEMU_QEMU_HARDDISK_drive-scsi2.bin 

properly says no header is detected on that drive and asks to confirm restore. The drive will however not mount.

 

However starting the array (unraid shows unmountable drive) and restoring the same backup file to /dev/mdX instead, then stopping/starting the array it will mount again.

 

Since on Unraid the first 32kB of every drive (/dev/sdX, /dev/nvmeX) is reserved and hidden from view (/dev/mdX starts at 0x8000 of /dev/sdX) one should really only access the mapped drives (also to maintain parity), so for array drives the backup probably needs to be done from this as well and it's a bit of a miracle that cryptsetup luksHeaderBackup /dev/sdX works on array drives despite the offset.

 

There is a difference between cache and array too, as can be seen from unraid mounting them:

Jan 20 13:46:30 Unraid2  emhttpd: shcmd (468): /usr/sbin/cryptsetup luksOpen /dev/md1 md1 --allow-discards
Jan 20 13:46:32 Unraid2  emhttpd: shcmd (472): /usr/sbin/cryptsetup luksOpen /dev/sdc1 sdc1 --allow-discards
Jan 20 13:46:34 Unraid2  emhttpd: shcmd (475): /usr/sbin/cryptsetup luksOpen /dev/sde1 sde1 --allow-discards

 

For cache the partition is mounted directly, but for array it's the mapped device, because it has to go through parity and trim those 32k...

 

More research/testing needed, but it seems to me that different handling is needed for array and cache, and for array it needs to be referenced to slots as well as drives, potentially making it a bit harder to script...

 

Edited by Kilrah
  • Thanks 1
Link to comment

Many @Kilrah,

 

Regarding the RESTORE PROCESS:

I read carefully your comment and I realize that indeed, the restore command is not correct. The "if cryptsetup luksDump $i &>/dev/null; then" part would indeed cause the restore to fail is the LUKS headers were completely removed (instead of just corrupted). Also, I realize that after a reboot, one drive may be assigned as another /dev/sdX, which would also cause the script to fail.

 

Regarding the BACKUP PROCESS:

I also understand that the backup script may not be correct, because of it using /dev/sdX instead of /dev/mdX. That part, I do not understand, especially your remark on parity. Indeed, I do not see why the "cryptsetup luksHeaderBackup" command would change parity.

 

I will edit my original post accordingly and update it when I can.

 

Meanwhile, any feedback or help would be greatly appreciated, especially from @limetech. Frankly, since Unraid offers to create encrypted partitions, I believe that backup/restore of LUKS headers should be embedded in Unraid directly.

 

Many thanks,

OP

Link to comment
1 hour ago, Kilrah said:

Manually running

cryptsetup luksHeaderRestore /dev/sdd --header-backup-file /boot/luks/luks-headers-backup-20230120132359/0QEMU_QEMU_HARDDISK_drive-scsi2.bin 

properly says no header is detected on that drive and asks to confirm restore. The drive will however not mount.

 

I really don't understand why the drive would not mount. Does that make sense to you ?

Link to comment

I requested an extension of my trial key and did some testing on my testing Unraid VM.

 

If I do this:

  • backup the LUKS header of a device /dev/sdX with "cryptsetup luksHeaderBackup /dev/sdc --header-backup-file QEMU_HARDDISK_QM00005.bin"
  • stop the array
  • erase the LUKS keys with "cryptsetup erase /dev/sdc"
  • restore the LUKS keys with "cryptsetup luksHeaderRestore /dev/sdc --header-backup-file QEMU_HARDDISK_QM00005.bin"
  • start the array

then indeed, the disk fails to mount.

 

If i do this:

  • backup the LUKS header of the parition /dev/sdX1 (notice the "1" added at the end) with "cryptsetup luksHeaderBackup /dev/sdc1 --header-backup-file QEMU_HARDDISK_QM00005.bin"
  • stop the array
  • erase the LUKS keys with "cryptsetup erase /dev/sdc1" (notice the "1" at the end again)
  • restore of the LUKS keys with "cryptsetup luksHeaderRestore /dev/sdc1 --header-backup-file QEMU_HARDDISK_QM00005.bin" (notice the "1" at the end once again)
  • start the array

then the disk mounts successfully.

 

I did this test with a 1 disk cache pool drive, and also with an array data drive.

I did a parity check after restoring the LUKS keys on the array data drive and no error were detected.

 

I think that clear one part of the problem.

 

Regarding the /dev/mdX suggestion, I notice that this "volume" is not available when the array is stopped, so I am unsure whether it should be used for the LUKS header backup/restore process (I could be wrong though).

 

Now I need to do the tests again with a complete deletion of the LUKS header (not just the LUKS keys), like you did, to see if the restore works, and if the parity is preserved.

Link to comment
2 hours ago, Opawesome said:

I do not see why the "cryptsetup luksHeaderBackup" command would change parity.

Backup won't but restore will, unless you're writing what was already there, or the restore corrected an error that went untracked in the first place.

 

2 hours ago, Opawesome said:

I really don't understand why the drive would not mount. Does that make sense to you ?

the headerbackup function skips to the start of the header automatically, the resulting bin starts with the LUKS marker. Restoring it to /dev/sdX restores it at the start of the disk so wipes the partition table. 

 

Looks like /dev/mdX does match /dev/sdX1 indeed. But writes to sdX will bypass parity, one should never write to an array disk without going through /dev/mdX.

Of course the array needs to be started.

 

image.png.78723fb82efa83c36f50d19dc462148e.png

Edited by Kilrah
Link to comment
17 hours ago, Kilrah said:

Backup won't but restore will, unless you're writing what was already there, or the restore corrected an error that went untracked in the first place.

 

the headerbackup function skips to the start of the header automatically, the resulting bin starts with the LUKS marker. Restoring it to /dev/sdX restores it at the start of the disk so wipes the partition table. 

 

Looks like /dev/mdX does match /dev/sdX1 indeed. But writes to sdX will bypass parity, one should never write to an array disk without going through /dev/mdX.

Of course the array needs to be started.

 

image.png.78723fb82efa83c36f50d19dc462148e.png

 

I opened a new General support thread to obtain additional info here and did some testing.

 

It seems that it is not possible to write to /dev/mdX if the array is not started, and starting the array with one bad LUKS header is not possible.

 

So I guess that in the scenario of a bad LUKS header preventing the start of the array, a rebuild of the parity would be needed.

 

Do you concur ?

Link to comment
7 hours ago, Opawesome said:

It seems that it is not possible to write to /dev/mdX if the array is not started, and starting the array with one bad LUKS header is not possible.

That is probably because the md type devices only get created on array start.    Have you actually tried starting in Maintenance mode as that does not mount the drives so I would be surprised if it checked any LUKS headers but it does create the md type devices.

Link to comment
16 minutes ago, itimpi said:

That is probably because the md type devices only get created on array start.    Have you actually tried starting in Maintenance mode as that does not mount the drives so I would be surprised if it checked any LUKS headers but it does create the md type devices.

Hi @itimpi, many thanks. I did indeed test that with @Kilrah in this thread, and the results are inconsistent. With the LUKS key missing on one drive, the array can start on @Kilrah's system, bit on mine the array fails to start.

However, with the LUKS header completely removed (not just the key), it seems that I can start the array.

Edited by Opawesome
Link to comment
  • 1 year later...

Hi,

 

I started to encrypt my drives and wanted to backup the luks header with your script.

 

Can I do this? Because in step 2 there are a lot "DO NOT USE" infos 😄

 

Are there any problems running the script? Did anyone made a restore which worked?

 

Thanks!

Edited by enJOyIT
Link to comment

The script to backup works fine.

 

It is just the automated restore command that might have issues. It just means that you will have to run the restore command manually, ie. identifying the right HD/nvme device and the right backed up header file with a command like:

 

cryptsetup luksHeaderRestore --header-backup-file <path to relevant LUKS header backup> <device / device partition>

 

IMPORTANT NOTE: If the disk is part of an array protected by parity, the <device / device partition> should be the managed partition device created upon starting the array in maintenance mode, i.e. "/dev/mdXp1", and not "/dev/sdY1". or else you will void the parity. For disk not protected by parity (e.g.: cache disk), pointing to the device partition (i.e.: "/dev/sdY1") seems to work better than pointing to the device itself (i.e.: "/dev/sdY")

 

see:

https://forums.unraid.net/topic/133972-backup-and-restore-of-luks-headers-on-array-data-drives/#comment-1217840

and

https://gitlab.com/cryptsetup/cryptsetup/-/blob/main/FAQ.md#6-backup-and-data-recovery

Edited by Opawesome
added note re. parity protected arrays
Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.