GraemeT Posted March 26, 2021 Share Posted March 26, 2021 I am having an issue with an encrypted array drive. I accidentally hot pulled the drive (stupid mistake as I was trying to pull an empty caddy and I managed to pull the wrong one!) What this has led to though is a walk through of what to do in the event of a failed disk.... and in my setup, a disk rebuild appears not to be working! I followed the unraid manual guidance for rebuilding back onto the same disk, which after 7 hours reported success.... however I am using encrypted XFS on the array and unraid reported that the disk was still unmountable after a rebuild. I ran cryptsetup to test if the encryption is OK... root@central:/mnt# cryptsetup -v isLuks /dev/sdb1 Command failed with code -1 (wrong or missing parameters). Apparently not! I am in the process of trying to rebuild for a second time: This time I have run a pre-clear on the disk for the first few hundred MB to wipe out any prior encryption or filesystem... the rebuild is currently on progress, but it is not looking promising as the luks headers are still not there according to the command. Does anyone have any ideas as to what I am doing wrong? This is all on Version: 6.9.1 Some screenshots to help show what I have done for the second attempt: State before bringing online (sdb is the disk that was in the array and I have run pre-clear on the first few hundred MB) Quote Link to comment
GraemeT Posted March 27, 2021 Author Share Posted March 27, 2021 (edited) I have narrowed the issue down to a corrupted LUKS header: root@central:/mnt/cache/recover# cryptsetup luksDump /dev/sdb1 --debug # cryptsetup 2.3.4 processing "cryptsetup luksDump /dev/sdb1 --debug" # Running command luksDump. # Locking memory. # Installing SIGINT/SIGTERM handler. # Unblocking interruption on signal. # Allocating context for crypt device /dev/sdb1. # Trying to open and read device /dev/sdb1 with direct-io. # Initialising device-mapper backend library. # Trying to load any crypt type from device /dev/sdb1. # Crypto backend (OpenSSL 1.1.1j 16 Feb 2021) initialized in cryptsetup library version 2.3.4. # Detected kernel Linux 5.10.21-Unraid x86_64. # Loading LUKS2 header (repair disabled). # Acquiring read lock for device /dev/sdb1. # Opening lock resource file /run/cryptsetup/L_8:17 # Verifying lock handle for /dev/sdb1. # Device /dev/sdb1 READ lock taken. # Trying to read primary LUKS2 header at offset 0x0. # Opening locked device /dev/sdb1 # Veryfing locked device handle (bdev) # LUKS2 header version 2 of size 16384 bytes, checksum sha256. # Checksum:770a0e992143ca7be43ae402b12b9df9f6c0356b09937647d0151453371397f2 (on-disk) # Checksum:245e32401668452f3fc9e86faf5e388f49417c108350338501b069a04e2ec6f6 (in-memory) # LUKS2 header checksum error (offset 0). # Trying to read secondary LUKS2 header at offset 0x4000. # Reusing open ro fd on device /dev/sdb1 # LUKS2 header version 2 of size 16384 bytes, checksum sha256. # Checksum:6991844e5efb3ce3606c1dbfc629dfbb4149475b0d19b865e43cfe5a00d07f3f (on-disk) # Checksum:8679968f9c132a9687e9a5a116e23c108a38fa4521e338c135f5942b217dac42 (in-memory) # LUKS2 header checksum error (offset 16384). # Trying to read secondary LUKS2 header at offset 0x8000. # Reusing open ro fd on device /dev/sdb1 # Trying to read secondary LUKS2 header at offset 0x10000. # Reusing open ro fd on device /dev/sdb1 # Trying to read secondary LUKS2 header at offset 0x20000. # Reusing open ro fd on device /dev/sdb1 # Trying to read secondary LUKS2 header at offset 0x40000. # Reusing open ro fd on device /dev/sdb1 # Trying to read secondary LUKS2 header at offset 0x80000. # Reusing open ro fd on device /dev/sdb1 # Trying to read secondary LUKS2 header at offset 0x100000. # Reusing open ro fd on device /dev/sdb1 # Trying to read secondary LUKS2 header at offset 0x200000. # Reusing open ro fd on device /dev/sdb1 # Trying to read secondary LUKS2 header at offset 0x400000. # Reusing open ro fd on device /dev/sdb1 # LUKS2 header read failed (-22). # Device /dev/sdb1 READ lock released. Device /dev/sdb1 is not a valid LUKS device. # Releasing crypt device /dev/sdb1 context. # Releasing device-mapper backend. # Closing read only fd for /dev/sdb1. # Unlocking memory. Command failed with code -1 (wrong or missing parameters). It appears that somehow both copies of the header have been corrupted in the exact same way (they are identical in the parts that are defined to be identical) : 00000000 4c 55 4b 53 ba be 00 02 00 00 00 00 00 00 40 00 |LUKS..........@.| 00000010 00 00 00 00 00 00 00 03 00 00 00 00 00 00 00 00 |................| 00000020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| * 00000040 00 00 00 00 00 00 00 00 73 68 61 32 35 36 00 00 |........sha256..| 00000050 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 00000060 00 00 00 00 00 00 00 00 60 63 7d 9c ab 52 b5 99 |........`c}..R..| 00000070 cf 5d a3 58 c0 06 3a 09 15 98 71 98 04 93 9c 5f |.].X..:...q...._| 00000080 be 70 9c ef b6 e1 af 4f 5b 7a 72 4b f0 38 d0 73 |.p.....O[zrK.8.s| 00000090 55 76 69 26 3f ce 8e 2f b1 9e 51 a2 47 79 8e 14 |Uvi&?../..Q.Gy..| 000000a0 3e 65 e0 83 8f d1 78 3a 34 38 39 64 62 65 34 37 |>e....x:489dbe47| 000000b0 2d 36 61 33 64 2d 34 38 61 33 2d 62 61 30 36 2d |-6a3d-48a3-ba06-| 000000c0 33 31 36 61 61 65 35 36 64 34 64 62 00 00 00 00 |316aae56d4db....| 000000d0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| * 000001c0 77 0a 0e 99 21 43 ca 7b e4 3a e4 02 b1 2b 9d f9 |w...!C.{.:...+..| 000001d0 f6 c0 35 6b 09 93 76 47 d0 15 14 53 37 13 97 f2 |..5k..vG...S7...| 000001e0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| * 00001000 7b 22 6b 65 79 73 6c 6f 74 73 22 3a 7b 22 30 22 |{"keyslots":{"0"| 00001010 3a 7b 22 74 79 70 65 22 3a 22 6c 75 6b 73 32 22 |:{"type":"luks2"| 00001020 2c 22 6b 65 79 5f 73 69 7a 65 22 3a 36 34 2c 22 |,"key_size":64,"| 00001030 61 66 22 3a 7b 22 74 79 70 65 22 3a 22 6c 75 6b |af":{"type":"luk| 00001040 73 31 22 2c 22 73 74 72 69 70 65 73 22 3a 34 30 |s1","stripes":40| 00001050 30 30 2c 22 68 61 73 68 22 3a 22 73 68 61 32 35 |00,"hash":"sha25| 00001060 36 22 7d 2c 22 61 72 65 61 22 3a 7b 22 74 79 70 |6"},"area":{"typ| 00001070 65 22 3a 22 72 61 77 22 2c 22 6f 66 66 73 65 74 |e":"raw","offset| 00001080 22 3a 22 33 32 37 36 38 22 2c 22 73 69 7a 65 22 |":"32768","size"| 00001090 3a 22 32 35 38 30 34 38 22 2c 22 65 6e 63 72 79 |:"258048","encry| 000010a0 70 74 69 6f 6e 22 3a 22 61 65 73 2d 78 74 73 2d |ption":"aes-xts-| 000010b0 70 6c 61 69 6e 36 34 22 2c 22 6b 65 79 5f 73 69 |plain64","key_si| 000010c0 7a 65 22 3a 36 34 7d 2c 22 6b 64 66 22 3a 7b 22 |ze":64},"kdf":{"| 000010d0 74 79 70 65 22 3a 22 61 72 67 6f 6e 32 69 22 2c |type":"argon2i",| 000010e0 22 74 69 6d 65 22 3a 34 2c 22 6d 65 6d 6f 72 79 |"time":4,"memory| 000010f0 22 3a 37 33 30 30 30 37 2c 22 63 70 75 73 22 3a |":730007,"cpus":| 00001100 34 2c 22 73 61 6c 74 22 3a 22 79 49 2b 56 77 44 |4,"salt":"yI+VwD| 00001110 7a 42 53 30 78 38 45 61 5a 41 4b 76 43 65 41 6a |zBS0x8EaZAKvCeAj| 00001120 63 54 44 4e 54 7a 56 51 36 46 63 35 53 6b 4f 75 |cTDNTzVQ6Fc5SkOu| 00001130 6c 47 57 7a 41 3d 22 7d 7d 7d 2c 22 74 6f 6b 65 |lGWzA="}}},"toke| 00001140 6e 73 22 3a 7b 7d 2c 22 73 65 67 6d 65 6e 74 73 |ns":{},"segments| 00001150 22 3a 7b 22 30 22 3a 7b 22 74 79 70 65 22 3a 22 |":{"0":{"type":"| 00001160 63 72 79 70 74 22 2c 22 6f 66 66 73 65 74 22 3a |crypt","offset":| 00001170 22 31 36 37 37 37 32 31 36 22 2c 22 73 69 7a 65 |"16777216","size| 00001180 22 3a 22 64 79 6e 61 6d 69 63 22 2c 22 69 76 5f |":"dynamic","iv_| 00001190 74 77 65 61 6b 22 3a 22 30 22 2c 22 65 6e 63 72 |tweak":"0","encr| 000011a0 79 70 74 69 6f 6e 22 3a 22 61 65 73 2d 78 74 73 |yption":"aes-xts| 000011b0 2d 70 6c 61 69 6e 36 34 22 2c 22 73 65 63 74 6f |-plain64","secto| 000011c0 72 5f 73 69 7a 65 22 3a 35 31 32 7d 7d 2c 22 64 |r_size":512}},"d| 000011d0 69 67 65 73 74 73 22 3a 7b 22 30 22 3a 7b 22 74 |igests":{"0":{"t| 000011e0 79 70 65 22 3a 22 70 62 6b 64 66 32 22 2c 22 6b |ype":"pbkdf2","k| 000011f0 65 79 73 6c 6f 74 73 22 3a 5b 22 30 22 5d 2c 22 |eyslots":["0"],"| 00001200 73 65 67 6d 65 6e 74 73 22 3a 5b 22 30 22 5d 2c |segments":["0"],| 00001210 22 68 61 73 68 23 3a 22 73 68 61 32 35 36 22 2c |"hash#:"sha256",| 00001220 22 69 74 6c 70 60 77 69 6f 6e 73 22 3a 34 30 32 |"itlp`wions":402| 00001230 30 36 2c 22 73 61 6c 74 22 0e 65 7b 47 39 2f 7b |06,"salt".e{G9/{| 00001240 78 54 02 40 34 5e 5d 5d 20 57 7b 56 5e 5d 69 5f |xT.@4^]] W{V^]i_| 00001250 30 5a 31 45 73 64 69 72 5d 19 50 5a 4c 29 02 20 |0Z1Esdir].PZL). | 00001260 4f 56 4c 7e 44 6b 3d 22 2c 22 64 69 67 65 73 74 |OVL~Dk=","digest| 00001270 22 3a 22 64 46 32 66 69 77 78 47 32 6d 44 6b 44 |":"dF2fiwxG2mDkD| 00001280 41 66 30 38 45 6b 71 6f 4e 6d 46 74 49 65 61 4a |Af08EkqoNmFtIeaJ| 00001290 56 52 73 4f 6b 70 78 4d 4e 49 49 4c 6d 6f 3d 22 |VRsOkpxMNIILmo="| 000012a0 7d 7d 2c 22 63 6f 6e 66 69 67 22 3a 7b 22 6a 73 |}},"config":{"js| 000012b0 6f 6e 5f 73 69 7a 65 22 3a 22 31 32 32 38 38 22 |on_size":"12288"| 000012c0 2c 22 6b 65 79 73 6c 6f 74 73 5f 73 69 7a 65 22 |,"keyslots_size"| 000012d0 3a 22 31 36 37 34 34 34 34 38 22 7d 7d 00 00 00 |:"16744448"}}...| 000012e0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| * 00004000 53 4b 55 4c ba be 00 02 00 00 00 00 00 00 40 00 |SKUL..........@.| 00004010 00 00 00 00 00 00 00 03 00 00 00 00 00 00 00 00 |................| 00004020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| * 00004040 00 00 00 00 00 00 00 00 73 68 61 32 35 36 00 00 |........sha256..| 00004050 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 00004060 00 00 00 00 00 00 00 00 fc a2 b5 4c c4 48 9e 04 |...........L.H..| 00004070 4c e6 a0 7b 4c c0 c1 f8 70 dc 81 67 07 06 4c 92 |L..{L...p..g..L.| 00004080 30 b8 eb ba 4c 3e b6 40 ae b7 bf d1 1e e6 04 2d |0...L>[email protected]| 00004090 38 0d 71 0f 6d 65 9c 66 69 e9 f6 c2 7d 68 58 fa |8.q.me.fi...}hX.| 000040a0 47 94 ba af a9 31 da 8d 34 38 39 64 62 65 34 37 |G....1..489dbe47| 000040b0 2d 36 61 33 64 2d 34 38 61 33 2d 62 61 30 36 2d |-6a3d-48a3-ba06-| 000040c0 33 31 36 61 61 65 35 36 64 34 64 62 00 00 00 00 |316aae56d4db....| 000040d0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| * 00004100 00 00 00 00 00 00 40 00 00 00 00 00 00 00 00 00 |......@.........| 00004110 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| * 000041c0 69 91 84 4e 5e fb 3c e3 60 6c 1d bf c6 29 df bb |i..N^.<.`l...)..| 000041d0 41 49 47 5b 0d 19 b8 65 e4 3c fe 5a 00 d0 7f 3f |AIG[...e.<.Z...?| 000041e0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| * 00005000 7b 22 6b 65 79 73 6c 6f 74 73 22 3a 7b 22 30 22 |{"keyslots":{"0"| 00005010 3a 7b 22 74 79 70 65 22 3a 22 6c 75 6b 73 32 22 |:{"type":"luks2"| 00005020 2c 22 6b 65 79 5f 73 69 7a 65 22 3a 36 34 2c 22 |,"key_size":64,"| 00005030 61 66 22 3a 7b 22 74 79 70 65 22 3a 22 6c 75 6b |af":{"type":"luk| 00005040 73 31 22 2c 22 73 74 72 69 70 65 73 22 3a 34 30 |s1","stripes":40| 00005050 30 30 2c 22 68 61 73 68 22 3a 22 73 68 61 32 35 |00,"hash":"sha25| 00005060 36 22 7d 2c 22 61 72 65 61 22 3a 7b 22 74 79 70 |6"},"area":{"typ| 00005070 65 22 3a 22 72 61 77 22 2c 22 6f 66 66 73 65 74 |e":"raw","offset| 00005080 22 3a 22 33 32 37 36 38 22 2c 22 73 69 7a 65 22 |":"32768","size"| 00005090 3a 22 32 35 38 30 34 38 22 2c 22 65 6e 63 72 79 |:"258048","encry| 000050a0 70 74 69 6f 6e 22 3a 22 61 65 73 2d 78 74 73 2d |ption":"aes-xts-| 000050b0 70 6c 61 69 6e 36 34 22 2c 22 6b 65 79 5f 73 69 |plain64","key_si| 000050c0 7a 65 22 3a 36 34 7d 2c 22 6b 64 66 22 3a 7b 22 |ze":64},"kdf":{"| 000050d0 74 79 70 65 22 3a 22 61 72 67 6f 6e 32 69 22 2c |type":"argon2i",| 000050e0 22 74 69 6d 65 22 3a 34 2c 22 6d 65 6d 6f 72 79 |"time":4,"memory| 000050f0 22 3a 37 33 30 30 30 37 2c 22 63 70 75 73 22 3a |":730007,"cpus":| 00005100 34 2c 22 73 61 6c 74 22 3a 22 79 49 2b 56 77 44 |4,"salt":"yI+VwD| 00005110 7a 42 53 30 78 38 45 61 5a 41 4b 76 43 65 41 6a |zBS0x8EaZAKvCeAj| 00005120 63 54 44 4e 54 7a 56 51 36 46 63 35 53 6b 4f 75 |cTDNTzVQ6Fc5SkOu| 00005130 6c 47 57 7a 41 3d 22 7d 7d 7d 2c 22 74 6f 6b 65 |lGWzA="}}},"toke| 00005140 6e 73 22 3a 7b 7d 2c 22 73 65 67 6d 65 6e 74 73 |ns":{},"segments| 00005150 22 3a 7b 22 30 22 3a 7b 22 74 79 70 65 22 3a 22 |":{"0":{"type":"| 00005160 63 72 79 70 74 22 2c 22 6f 66 66 73 65 74 22 3a |crypt","offset":| 00005170 22 31 36 37 37 37 32 31 36 22 2c 22 73 69 7a 65 |"16777216","size| 00005180 22 3a 22 64 79 6e 61 6d 69 63 22 2c 22 69 76 5f |":"dynamic","iv_| 00005190 74 77 65 61 6b 22 3a 22 30 22 2c 22 65 6e 63 72 |tweak":"0","encr| 000051a0 79 70 74 69 6f 6e 22 3a 22 61 65 73 2d 78 74 73 |yption":"aes-xts| 000051b0 2d 70 6c 61 69 6e 36 34 22 2c 22 73 65 63 74 6f |-plain64","secto| 000051c0 72 5f 73 69 7a 65 22 3a 35 31 32 7d 7d 2c 22 64 |r_size":512}},"d| 000051d0 69 67 65 73 74 73 22 3a 7b 22 30 22 3a 7b 22 74 |igests":{"0":{"t| 000051e0 79 70 65 22 3a 22 70 62 6b 64 66 32 22 2c 22 6b |ype":"pbkdf2","k| 000051f0 65 79 73 6c 6f 74 73 22 3a 5b 22 30 22 5d 2c 22 |eyslots":["0"],"| 00005200 73 65 67 6d 65 6e 74 73 22 3a 5b 22 30 22 5d 2c |segments":["0"],| 00005210 22 68 61 73 68 23 3a 22 73 68 61 32 35 36 22 2c |"hash#:"sha256",| 00005220 22 69 74 6c 70 60 77 69 6f 6e 73 22 3a 34 30 32 |"itlp`wions":402| 00005230 30 36 2c 22 73 61 6c 74 22 0e 65 7b 47 39 2f 7b |06,"salt".e{G9/{| 00005240 78 54 02 40 34 5e 5d 5d 20 57 7b 56 5e 5d 69 5f |xT.@4^]] W{V^]i_| 00005250 30 5a 31 45 73 64 69 72 5d 19 50 5a 4c 29 02 20 |0Z1Esdir].PZL). | 00005260 4f 56 4c 7e 44 6b 3d 22 2c 22 64 69 67 65 73 74 |OVL~Dk=","digest| 00005270 22 3a 22 64 46 32 66 69 77 78 47 32 6d 44 6b 44 |":"dF2fiwxG2mDkD| 00005280 41 66 30 38 45 6b 71 6f 4e 6d 46 74 49 65 61 4a |Af08EkqoNmFtIeaJ| 00005290 56 52 73 4f 6b 70 78 4d 4e 49 49 4c 6d 6f 3d 22 |VRsOkpxMNIILmo="| 000052a0 7d 7d 2c 22 63 6f 6e 66 69 67 22 3a 7b 22 6a 73 |}},"config":{"js| 000052b0 6f 6e 5f 73 69 7a 65 22 3a 22 31 32 32 38 38 22 |on_size":"12288"| 000052c0 2c 22 6b 65 79 73 6c 6f 74 73 5f 73 69 7a 65 22 |,"keyslots_size"| 000052d0 3a 22 31 36 37 34 34 34 34 38 22 7d 7d 00 00 00 |:"16744448"}}...| 000052e0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| The corruption appears to start at 1210 "hash#: should be "hash": and continues in the following few bytes. Most of it is fixable, but it appears that the salt was badly affected. It may be possible to write something to brute force that based on the sha256 for the header, but that assumes that non of the other data has corruption. I am not concerned about the data, but does anyone have any ideas how both the LUKS headers may have been corrupted in this subtle way? I do not think it was because I accidentally pulled the drive. If I don't get any other suggestions today, I will reformat and rebuild the disk. I will then pull the disk in the same way and see if I can replicate the issue. Edited March 27, 2021 by GraemeT Fixed typo Quote Link to comment
GraemeT Posted April 7, 2021 Author Share Posted April 7, 2021 I pulled the drive and performed a rebuild onto the same disk and it worked. Given the type of corruption and the exact same corruption in both LUKS Headers I would assume this issue was due to some LUKS activity that went wrong. The rebuild process seems to not be the cause. As everything is working fine now, and I am unable to recrate the issue, I am considering this closed. Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.