While still being in the release candidate phase of unRAID 6.4 and wanting to learn more about unRAID I decided to do an experiment involving parity drive replacement.

 

Assumption: The parity drives are used as unstructured raw devices containing the parity bits calculated from the data drives.

 

Result: This is not the case, parity drives are partitioned, they seem to contain one large partition without file system which contains the parity bits. I could have seen this easily by clicking on the disk ID on the "Main" tab where it states "Partition format: GPT: 4k-aligned" ...

 

Details: I was hoping for an unstructured device because this would allow for a parity drive size upgrade while always being fully protected, which is not the case with the standard parity upgrade procedure. The procedure I tried was as follows (I don't include preparation work like testing new drives, checking parity and SMART before starting, etc.):

1. with the future parity drives installed, but unassigned, copy full content of current parity drives to future parity drives using dd
2. fill the reminder of future parity drives with zeroes using dd
3. stop the array, unassigned current parity drives, assign new drives as parity drives
4. do a parity check.

 

Alas the parity check result in 80 parity errors clustered around four locations on the drives:

Dec 20 00:32:53 Tower kernel: md: recovery thread: PQ incorrect, sector=214xxxxxxx

Dec 20 07:07:56 Tower kernel: md: recovery thread: PQ incorrect, sector=644xxxxxxx

Dec 20 10:25:01 Tower kernel: md: recovery thread: PQ incorrect, sector=858xxxxxxx

Dec 20 21:15:56 Tower kernel: md: recovery thread: PQ incorrect, sector=1562xxxxxxx

When seeing that I immediately suspected partitioning as root cause of the error. Partition tables are written to the beginning and end of each drive, therefore the way I copied parity drives would result in parity errors with the new larger drives at the position where the second partition table was on the smaller previous drives.

In syslog I could also find the partitioning commands:

Dec 19 21:12:29 Tower emhttpd: writing GPT on disk (sdd), with partition 1 byte offset 32K, erased: 0
Dec 19 21:12:29 Tower emhttpd: shcmd (752): sgdisk -Z /dev/sdd
Dec 19 21:12:30 Tower emhttpd: shcmd (753): sgdisk -o -a 8 -n 1:32K:0 /dev/sdd
Dec 19 21:12:32 Tower emhttpd: writing GPT on disk (sdg), with partition 1 byte offset 32K, erased: 0
Dec 19 21:12:32 Tower emhttpd: shcmd (755): sgdisk -Z /dev/sdg
Dec 19 21:12:33 Tower emhttpd: shcmd (756): sgdisk -o -a 8 -n 1:32K:0 /dev/sdg

I don't yet understand why I am seeing additional clusters of errors and how the sectors are counted. The new parity drives are 12TB drives having 4096 bytes logical/physical sectors, so a sector 1562xxxxxxx doesn't even exist. The old parity drives were 8TB drives, just like all of the array drives.

 

Can somebody help to answer the following questions:

1. How are the sectors counted when logging parity errors?
2. Any idea as why there are clusters of errors in addition to those where the second partition table was?
3. What is the motivation to partition the parity drives and thereby not protect the partition tables on the data drives?
4. The man page of sgdisk claims that with "sgdisk -a 8" as used by unRAID, partitions would be aligned to "8 x sector size". My impression however is that partitions are aligned to 8 x 512 bytes which for my disks is not the same. Am I right?

The proper way to upgrade a parity disk is to assign it and then start a parity sync operation.  This ensures that if any writes occur to any of the other devices during parity sync, the new parity device will be written as well.

 

Array devices in unRAID all have the same layout: they start 32K from the beginning of the device and extend to the "largest size possible".

 

For 512-byte sectors, 32K => 64 sectors.

For 4096-byte sectors, 32K => 8 sectors.

Thus for both cases, telling sgdisk that partition aligns on a 8-sector boundary is correct and prevents it from writing a confusing message to syslog.

 

All devices, including parity devices, have a valid partition table structure written to them.  These structures are outside the data storage range, ie, within the first 32K or beyond the "largest size possible" area.

 

Parity devices are given a valid partition layout for two reasons:

1. So that at least MBR is "valid" from linux point of view.  These days probably doesn't much matter, but in past if you had the right random data in a partition table, or remnants of a partition table because device was used in different system, linux would get very confused.  These problems are avoided simply by initializing the device properly.

2. Realize that in a 2-device configuration: Parity + Disk1 or 3-device: Parity1 + Parity2 + Disk1, all devices have the exact same data, that is, either a 2-way or 3-way mirror.  Hence you can dismantle such an unRAID array and use the data on any of the devices.  This is by design.

 

Lastly, here's what "largest size possible" means.  Define the "size of the device" (that is, the size of partition 1) such that:

1. The size is a multiple of 4K, and

2. In the case of GPT, we take into account the sectors used to store the "backup" GPT header at the end of the device.

 

Hope this helps.

40 minutes ago, tstor said:

How are the sectors counted when logging parity errors?

 

The linux block layer always refers to 512-byte sectors, no matter the physical sector size.

6 minutes ago, limetech said:

Hope this helps.

 

Very helpful indeed and thanks a lot for taking the time.

As I wrote, I knew even before starting that this is not the official way to replace a parity disk, but I was curious to see, whether there would be some support data be written in addition to the parity itself. I wouldn't have done that once my array becomes fully productive.