Hi all,

 

In the .zip archive file attached to this post is a script I have written in order to backup LUKS headers of encrypted devices attached to Unraid.

 

1. WHY?

Backing up LUKS headers is a paramount precautionary measure given the fact that data corruption in LUKS header may lead to losing all the data present on the device. Please note that in this scenario, knowing the passphrase (or having a copy of the key file used to store the passphrase) is irrelevant and will not help you recover the data. This is because the passphrase is not used in deriving the encryption master key, but only used in decrypting the encryption master key, the latter being randomly generated on each device upon initial creation of the LUKS header. 

 

2. HOW?

 

To backup your LUKS headers, just run the attached script as root. The script is not interactive, therefore you can use it with the wonderful User.Script plugin by @Squid  (see: here).

 

3. WHEN?

 

LUKS headers only need to be backed up upon a change in one of the headers. A change should occur upon events such as:

 

• any drive is formatted (or reformatted) as encrypted (this includes the addition of a new encrypted drive)
• a passphrase (or the key file used to store the passphrase) is added to, changed in or removed from any of the 8 different key slots provided by LUKS (the Unraid GUI allows you to create only one passphrase for the entire array, but the underlying kernel module does allow 8 different passphrases for each encrypted device)

 

Therefore, there is no need to schedule the backup to any particular period. A scheduled backup should even be avoided if you elect to copy the backed up LUKS headers to the Unraid flash drive, as it is considered good practice to keep the read/write operations of the flash drive to a minimum.

 

4. HOW DO I RESTORE LUKS HEADERS FROM A PREVIOUS BACKUP?

 

As of now, the safest way I have found to restore LUKS headers from a previous backup is to run the command below manually for each drive that needs to have its LUKS header restored (don't forget to replace the <path to relevant LUKS header backup> and <device / device partition> with the correct information for your use case):

 

cryptsetup luksHeaderRestore --header-backup-file <path to relevant LUKS header backup> <device / device partition>

 

IMPORTANT: if the <device / device partition> is assigned to an array protected by parity, ensure that you are pointing to the managed partition device, i.e.: "/dev/mdXp1" (start the array in maintenance mode first) and not to the actual "dev/sdY1" partition device. Indeed, writes to /dev/sdY1 will bypass parity but writes to /dev/mdXp1 do not.

 

5. WORK IN PROGRESS

 

I am trying to write a command that would restore all LUKS header at once. However, I need to find a way to reconcile (in CLI) each "/dev/mdXp1" device (i.e.: the device that is created after starting the array and which needs to be written on for the restore operation in order to preserve parity) to the corresponding "/dev/sdY" device (i.e.: the actual device which contains the "ID_SERIAL" used to name the LUKS header backup .bin file for that disk).

 

I asked the question here but never got an answer.

 

The command would be something like (note that you should not run it as is because it is not working in its current state):

 

# DO NOT USE / DO NOT USE / DO NOT USE
#cd "/directory/where/the/backed/up/LUKS/headers/are/stored" && for i in {/dev/sd*,/dev/nvme*}; do if cryptsetup luksDump $i &>/dev/null; then cryptsetup luksHeaderRestore <WIP: '$i' cannot be used here because it would bypass parity> --header-backup-file "`udevadm info --query=all --name=$i | sed -n 's/.*ID_SERIAL=//p'`.bin"; fi; done
# DO NOT USE / DO NOT USE / DO NOT USE

 

5. ANYTHING ELSE?

 

Feel free to report any issues and make any suggestions.

 

This script is based on exchanges from the following threads, which you can refer to: 

• thread 1
• thread 2

as well as on the official LUKS page :

• https://gitlab.com/cryptsetup/cryptsetup/-/blob/main/FAQ.md#6-backup-and-data-recovery

 

Best,

OP

 

 

unraid_backup-luks-headers-v0.2a.zip

Edited February 20, 20242 yr by Opawesome
Added new section 3: "WHEN?"; Added clarification that the script is in the attached .zip archive file; Updated script tp fix error when disk's SERIAL_ID contains space characters; Added noew section 4 HOW DO I RESTORE LUKS HEADERS FROM A PREVIOUS BACKUP

Just a note of caution. Only write to the USB when absolutely necessary. Unraid takes great pains to limit the data written to the boot USB, and needlessly scheduling this script to run periodically could be counterproductive.

I recommend manually running this when you make a change, don't schedule it to make it convenient.

2 hours ago, jonathanm said:

Just a note of caution. Only write to the USB when absolutely necessary. Unraid takes great pains to limit the data written to the boot USB, and needlessly scheduling this script to run periodically could be counterproductive.

I recommend manually running this when you make a change, don't schedule it to make it convenient.

 

Thank you @jonathanm. Very relevant comment. I added a new section to the initial post: "3. WHEN?", in order to address this issue.

 

Best,

OP

Thx 4 this great script! Stumbled upon it when i was looking for a backup method. Really simple and awesome script! Thank you very much! 

Hi @Opawesome,

 

I've setup my first unraid server recently, so a total newbie at unraid. I've got my array/cache set to XFS - encrypted, with the hopes of should a drive ever fail I can RMA it without having to worry about data that might be on the drives.

 

When I run the script however, I get the following (I've hashed out the last few digits of S/N, and I get this message for all drives):

"Device ST4000VN008-2DR166_########.bin does not exist or access denied.
Header backup file does not contain compatible LUKS header."

 

Could you perhaps expand on the "How" section a bit more? As I'm missing something, but not sure what.

Have you managed to test the Restore functionality by any chance?

 

Regards,

Grimlious

7 hours ago, Grimlious said:

When I run the script however, I get the following (I've hashed out the last few digits of S/N, and I get this message for all drives):

"Device ST4000VN008-2DR166_########.bin does not exist or access denied.
Header backup file does not contain compatible LUKS header."

 

Hi @Grimlious,

 

Reading that error, it looks like you are trying to run the "restore" command i mentionned in my original post and not the script inside the .zip archive attached to it.

 

Can you confirm that you downloaded the attached .zip file and run the script contained therein ?

 

With regards,

OP

Edited February 20, 20224 yr by Opawesome

Hi OP.

 

Thanks, I didn't even see the attached file. For some reason I thought the script would do both.

 

Will try this when I get home.

 

Regards, 

Grimlious

Edited February 22, 20224 yr by Grimlious

Nice script, thank you!  but i think that it needs some little rework.

It's not possible to backup the luks headers of several encrypted nvme's.

 

I have 3 Samsung Nvme and its only generating one file named Samsung in the backup location for them. Also i received the following error during the process ->

 

Script location: /tmp/user.scripts/tmpScripts/unraid_backup-luks-headers/script
Note that closing this window will abort the execution of this script
This script will backup the LUKS headers of all LUKS encrypted devices which are present on this host (Sokrates) to the following directory: /mnt/disks/2116E598E8E4/luksheaders/.
This script will also copy the backed up LUKS headers to the Unraid flash drive, under the following directory: /boot/config/luks
Now starting backup of LUKS headers...
Requested header backup file Samsung already exists.
Requested header backup file Samsung already exists.
Backup of LUKS headers failed. Now exiting...
/tmp/user.scripts/tmpScripts/unraid_backup-luks-headers/script: line 80: return: can only `return' from a function or sourced script
Now setting proper owner and permissions for backed up LUKS headers...
Now starting copy of backed up LUKS headers to the Unraid flash drive...
Copy of backed up LUKS headers to the Unraid flash drive completed successfully.
Now exiting...

 

 

This worked for me, backed up and copyed this for my nvmes manually, with this example:

cryptsetup luksHeaderBackup /dev/nvme1n1p1 --header-backup-file=/backuplocation/file.bin

 

Edited August 14, 20223 yr by T-Birth

On 8/14/2022 at 6:57 PM, T-Birth said:

Nice script, thank you!  but i think that it needs some little rework.

It's not possible to backup the luks headers of several encrypted nvme's.

 

I have 3 Samsung Nvme and its only generating one file named Samsung in the backup location for them. Also i received the following error during the process [...]

 

Thank you @T-Birth for reporting the error. Indeed, I also have this error since I upgraded Unraid to a version above 6.8.3.

 

I have updated the script and backup command to fix it.

 

It turned out that I needed to add some double quotes because some disks' SERIAL_ID now have space characters in them.

 

I successfully tested the backup script (but not the restore command) on Unraid 6.10.3.

I don't know if anyone ever tried the restore method until now, but I just did and here are some notes, as far as I can see the described method can't work at least on array drives:

 

- Backed up with the script, with array running

- Using info from https://wiki.archlinux.org/title/Dm-crypt/Drive_preparation#Wipe_LUKS_header I erased all keys on an array drive and the header

- Tried restoring with the restore command, both with array stopped and started

 

This only asked me to confirm overwriting of existing headers on the other drives, not about the one that had it wiped. Seems the "if cryptsetup luksDump $i &>/dev/null; then" purposefully would not include any disk that does not already have a valid header, seems the opposite of what is wanted

 

Manually running

cryptsetup luksHeaderRestore /dev/sdd --header-backup-file /boot/luks/luks-headers-backup-20230120132359/0QEMU_QEMU_HARDDISK_drive-scsi2.bin 

properly says no header is detected on that drive and asks to confirm restore. The drive will however not mount.

 

However starting the array (unraid shows unmountable drive) and restoring the same backup file to /dev/mdX instead, then stopping/starting the array it will mount again.

 

Since on Unraid the first 32kB of every drive (/dev/sdX, /dev/nvmeX) is reserved and hidden from view (/dev/mdX starts at 0x8000 of /dev/sdX) one should really only access the mapped drives (also to maintain parity), so for array drives the backup probably needs to be done from this as well and it's a bit of a miracle that cryptsetup luksHeaderBackup /dev/sdX works on array drives despite the offset.

 

There is a difference between cache and array too, as can be seen from unraid mounting them:

Jan 20 13:46:30 Unraid2  emhttpd: shcmd (468): /usr/sbin/cryptsetup luksOpen /dev/md1 md1 --allow-discards
Jan 20 13:46:32 Unraid2  emhttpd: shcmd (472): /usr/sbin/cryptsetup luksOpen /dev/sdc1 sdc1 --allow-discards
Jan 20 13:46:34 Unraid2  emhttpd: shcmd (475): /usr/sbin/cryptsetup luksOpen /dev/sde1 sde1 --allow-discards

 

For cache the partition is mounted directly, but for array it's the mapped device, because it has to go through parity and trim those 32k...

 

More research/testing needed, but it seems to me that different handling is needed for array and cache, and for array it needs to be referenced to slots as well as drives, potentially making it a bit harder to script...

 

Edited January 20, 20233 yr by Kilrah

Many @Kilrah,

 

Regarding the RESTORE PROCESS:

I read carefully your comment and I realize that indeed, the restore command is not correct. The "if cryptsetup luksDump $i &>/dev/null; then" part would indeed cause the restore to fail is the LUKS headers were completely removed (instead of just corrupted). Also, I realize that after a reboot, one drive may be assigned as another /dev/sdX, which would also cause the script to fail.

 

Regarding the BACKUP PROCESS:

I also understand that the backup script may not be correct, because of it using /dev/sdX instead of /dev/mdX. That part, I do not understand, especially your remark on parity. Indeed, I do not see why the "cryptsetup luksHeaderBackup" command would change parity.

 

I will edit my original post accordingly and update it when I can.

 

Meanwhile, any feedback or help would be greatly appreciated, especially from @limetech. Frankly, since Unraid offers to create encrypted partitions, I believe that backup/restore of LUKS headers should be embedded in Unraid directly.

 

Many thanks,

OP

1 hour ago, Kilrah said:

Manually running

cryptsetup luksHeaderRestore /dev/sdd --header-backup-file /boot/luks/luks-headers-backup-20230120132359/0QEMU_QEMU_HARDDISK_drive-scsi2.bin 

properly says no header is detected on that drive and asks to confirm restore. The drive will however not mount.

 

I really don't understand why the drive would not mount. Does that make sense to you ?

I requested an extension of my trial key and did some testing on my testing Unraid VM.

 

If I do this:

• backup the LUKS header of a device /dev/sdX with "cryptsetup luksHeaderBackup /dev/sdc --header-backup-file QEMU_HARDDISK_QM00005.bin"
• stop the array
• erase the LUKS keys with "cryptsetup erase /dev/sdc"
• restore the LUKS keys with "cryptsetup luksHeaderRestore /dev/sdc --header-backup-file QEMU_HARDDISK_QM00005.bin"
• start the array

then indeed, the disk fails to mount.

 

If i do this:

• backup the LUKS header of the parition /dev/sdX1 (notice the "1" added at the end) with "cryptsetup luksHeaderBackup /dev/sdc1 --header-backup-file QEMU_HARDDISK_QM00005.bin"
• stop the array
• erase the LUKS keys with "cryptsetup erase /dev/sdc1" (notice the "1" at the end again)
• restore of the LUKS keys with "cryptsetup luksHeaderRestore /dev/sdc1 --header-backup-file QEMU_HARDDISK_QM00005.bin" (notice the "1" at the end once again)
• start the array

then the disk mounts successfully.

 

I did this test with a 1 disk cache pool drive, and also with an array data drive.

I did a parity check after restoring the LUKS keys on the array data drive and no error were detected.

 

I think that clear one part of the problem.

 

Regarding the /dev/mdX suggestion, I notice that this "volume" is not available when the array is stopped, so I am unsure whether it should be used for the LUKS header backup/restore process (I could be wrong though).

 

Now I need to do the tests again with a complete deletion of the LUKS header (not just the LUKS keys), like you did, to see if the restore works, and if the parity is preserved.

2 hours ago, Opawesome said:

I do not see why the "cryptsetup luksHeaderBackup" command would change parity.

Backup won't but restore will, unless you're writing what was already there, or the restore corrected an error that went untracked in the first place.

 

2 hours ago, Opawesome said:

I really don't understand why the drive would not mount. Does that make sense to you ?

the headerbackup function skips to the start of the header automatically, the resulting bin starts with the LUKS marker. Restoring it to /dev/sdX restores it at the start of the disk so wipes the partition table. 

 

Looks like /dev/mdX does match /dev/sdX1 indeed. But writes to sdX will bypass parity, one should never write to an array disk without going through /dev/mdX.

Of course the array needs to be started.

 

Edited January 20, 20233 yr by Kilrah

17 hours ago, Kilrah said:

Backup won't but restore will, unless you're writing what was already there, or the restore corrected an error that went untracked in the first place.

 

the headerbackup function skips to the start of the header automatically, the resulting bin starts with the LUKS marker. Restoring it to /dev/sdX restores it at the start of the disk so wipes the partition table. 

 

Looks like /dev/mdX does match /dev/sdX1 indeed. But writes to sdX will bypass parity, one should never write to an array disk without going through /dev/mdX.

Of course the array needs to be started.

 

 

I opened a new General support thread to obtain additional info here and did some testing.

 

It seems that it is not possible to write to /dev/mdX if the array is not started, and starting the array with one bad LUKS header is not possible.

 

So I guess that in the scenario of a bad LUKS header preventing the start of the array, a rebuild of the parity would be needed.

 

Do you concur ?

7 hours ago, Opawesome said:

It seems that it is not possible to write to /dev/mdX if the array is not started, and starting the array with one bad LUKS header is not possible.

That is probably because the md type devices only get created on array start.    Have you actually tried starting in Maintenance mode as that does not mount the drives so I would be surprised if it checked any LUKS headers but it does create the md type devices.

16 minutes ago, itimpi said:

That is probably because the md type devices only get created on array start.    Have you actually tried starting in Maintenance mode as that does not mount the drives so I would be surprised if it checked any LUKS headers but it does create the md type devices.

Hi @itimpi, many thanks. I did indeed test that with @Kilrah in this thread, and the results are inconsistent. With the LUKS key missing on one drive, the array can start on @Kilrah's system, bit on mine the array fails to start.

However, with the LUKS header completely removed (not just the key), it seems that I can start the array.

Edited January 21, 20233 yr by Opawesome

Hi,

 

I started to encrypt my drives and wanted to backup the luks header with your script.

 

Can I do this? Because in step 2 there are a lot "DO NOT USE" infos 😄

 

Are there any problems running the script? Did anyone made a restore which worked?

 

Thanks!

Edited February 2, 20242 yr by enJOyIT

The script to backup works fine.

 

It is just the automated restore command that might have issues. It just means that you will have to run the restore command manually, ie. identifying the right HD/nvme device and the right backed up header file with a command like:

 

cryptsetup luksHeaderRestore --header-backup-file <path to relevant LUKS header backup> <device / device partition>

 

IMPORTANT NOTE: If the disk is part of an array protected by parity, the <device / device partition> should be the managed partition device created upon starting the array in maintenance mode, i.e. "/dev/mdXp1", and not "/dev/sdY1". or else you will void the parity. For disk not protected by parity (e.g.: cache disk), pointing to the device partition (i.e.: "/dev/sdY1") seems to work better than pointing to the device itself (i.e.: "/dev/sdY")

 

see:

https://forums.unraid.net/topic/133972-backup-and-restore-of-luks-headers-on-array-data-drives/#comment-1217840

and

https://gitlab.com/cryptsetup/cryptsetup/-/blob/main/FAQ.md#6-backup-and-data-recovery

Edited February 20, 20242 yr by Opawesome
added note re. parity protected arrays

Thanks for this script!  Giving this a little bump since I think it's an important that people who use encrypted volumes perform.

 

I noticed when I dumped all of my headers, I had bin files of different sizes. Turns out, my disks in the array have a mixture of header version 1 and version 2 - where a version 1 header dump is about 1.1MB and version 2 is about 16MB. This must have been from disks that were formatted in older versions of unraid. I was completely unaware of this, however to me it seems to be inconsequential in daily operations.

 

For users who have not read-between-the-lines in the script or know how to work with cryptsetup, you can print out the headers and see things like what ciphers and hashes are in use, header version, and other information about your disks that you don't normally see in the GUI. Run the following in the unraid console or SSH to present a human readable version of the header to you:

cryptsetup luksDump FileName.bin

 

You can also do this directly to mounted disks instead of an image. For example, change the filename to "/dev/sdb1". I suppose you can perform this to manually verify your backups were successful.

Running Unraid 7.0 now, and only one disk (my cache disk) is detected as a valid LUKS device according to cryptsetup. This is with the array started and disks mounted. Previously I seem to remember being able to do this while the array was mounted?

 

root@server:/# cryptsetup luksDump /dev/sda
Device /dev/sda is not a valid LUKS device.
root@server:/# cryptsetup luksDump /dev/sda1
Device /dev/sda1 is not a valid LUKS device.

 

Same situation for the /dev/md devices and the /dev/dm- devices.

Any ideas?

hi I ran this script but script shows the success message but its not creating any zip backup file in the destination folder just creating empty folders. I am getting this massage

 

Quote

Backup of LUKS headers completed successfully.
Now setting proper owner and permissions for backed up LUKS headers...
chmod: missing operand after ‘666’
Try 'chmod --help' for more information.
Now starting copy of backed up LUKS headers to the Unraid flash drive...
Copy of backed up LUKS headers to the Unraid flash drive completed successfully.
Now exiting..."

 

 

I am using unraid 7

It seems like this script no longer works for array devices in unraid 7.

The tl;dr is that for the array, the /dev/md* devices should be targeted instead. For cache pool (at least for btrfs), things still seems to work.

Just changing the pattern to include /dev/md* wont' work, as this fails when udevadm gathers data about the device. I'll see if there's a way to correlate the md devices with the real device ID sometime later.

I'm wondering if there's anything preventing using lsblk to query for devices and filtering with isLuks, e.g.

for device in `lsblk -o path -ln`; do 
	if cryptsetup isLuks $device; then 
		...

This would help the script being less susceptible to arbitrary changes to where devices are located.

Below is the process I went through

I was curious about whether the /dev/md* devices would yield the same result as /dev/sd* did previously.

# stop the array first

## cache pool (btrfs)
cryptsetup luksHeaderBackup /dev/sdb1 --header-backup-file luks-sdb1.bin  
cryptsetup luksHeaderBackup /dev/sdc1 --header-backup-file luks-sdc1.bin
cryptsetup luksHeaderBackup /dev/nvme0n1p1 --header-backup-file luks-nvme0n1p1.bin

## array (xfs)
cryptsetup luksHeaderBackup /dev/sdd5 --header-backup-file luks-sdd5.bin
cryptsetup luksHeaderBackup /dev/sde1 --header-backup-file luks-sde1.bin
cryptsetup luksHeaderBackup /dev/sdf1 --header-backup-file luks-sdf1.bin

# after starting the array again
for md in /dev/md*; do 
	cryptsetup luksHeaderBackup $md --header-backup-file "luks-`basename $md`.bin";
done

I confirmed that the files are identical using cmp, the hashes just made it easy to confirm if any were close enough to bother checking.

shasum luks*.bin | sort  
5ad508c36ea8373a09e5abba49a8685f89464b33  luks-nvme0n1p1.bin  
60c879daae36ccdef2f02413a74f03ccb75104a3  luks-sdb1.bin  
814856d2565da05f65a7b244a3bc622b47972cec  luks-md2p1.bin  
814856d2565da05f65a7b244a3bc622b47972cec  luks-sde1.bin  
91e68de3533b697c4e7542c42661af3312968ecf  luks-md1p1.bin  
91e68de3533b697c4e7542c42661af3312968ecf  luks-sdd5.bin  
a44b321e6b3e421c35fba5b93a2d8ae6130e9504  luks-sdc1.bin  
e5585bf7ddddfdd8c4160073b6f8f1c1a31581c7  luks-md3p1.bin  
e5585bf7ddddfdd8c4160073b6f8f1c1a31581c7  luks-sdf1.bin

This matches the six encrypted disks in my server. I feel recently confident about taking the luks headers from the md devices.

Hey everyone, joining in this conversation as I transition to unRAID here in 2026. I would feel better about running encrypted drives, and understand the necessity of backing up the data; but like many of you, would like to also back up LUKS header for easier restore.

Couple questions about LUKS within unRAID first, with a failing drive in a parity protected xfs-encrypted array, a drive swap as normal can restore data (assuming 1 drive in a 1 parity protected array). Restoring of LUKS Header is specifically if you notice your passphrase no longer unlocks drives, but SMART results show drive is healthy, therefore, we assume LUKS header corruption, correct?

I do agree that because unRAID offers encryption, there should be a more "baked-in" easier way of backing up and restoring LUKS Headers, alas, I'm new here so looking to learn how to get this done from you fine folks.

Based on what Opawesome mentioning, sounds like dev paths line up with what docs mention here for data recovery in unRAID. "Always use /dev/mdXp1 for array drives to maintain valid parity." Okay.
Based on what @polakkenak mentioned, you were able to successfully backup headers of all parity protected drives (including parity drive)? Were you able to restore?

So, where are we at in unRAID 7.* for backing up and restorings LUKS Headers
cryptsetup luksHeaderBackup <device> --header-backup-file <file>
cryptsetup luksHeaderRestore <device> --header-backup-file <file>

Will these terminal commands get the job done (I assume both backup and restore is with the array stopped)?

I'm looking to do a LUKS Header backup, delete, and restore on a drive before putting data on it to show myself I'm competent enough to run an encrypted array heh. Any up-to-date help is greatly appreciated!

Edited April 1Apr 1 by mikescott
grammar