[SOLVED] Reading raw block devices and manually computing parity

[nick5429]

Disclaimer: very advanced topic. 99.9% of users will have no use for this. Intentionally posting code that doesn't compile lest someone try to do something they shouldn't.

 

 

So I've been poking around a bit reading the raw block devices (/dev/sdX) via some C code I wrote a) for intellectual curiosity and b) to help further diagnose some of the weirdness I've reported in other threads

 

I have 4 data disks (sda, sdb, sdc, sdd) and 1 parity (sde).

 

Basic procedure (pseudocode mixed with actual C syscall names)

//note: using lseek(2), open(2), etc rather than fseek(3), open(3), etc because they allow access to the entire block device with some special switches. Tried the latter with the same behavior.
//simplified to just read 1 byte instead of a large buffer
uint8_t diskBuf[5]; //one for each disk
int parIdx=4; //parity lives at index 4 in the diskBuf array
uint8_t calcParity;
uint8_t expectedParity;
int fd;
off_t offset=511;
devices = {sda, sdb, sdc, sdd, sde}

for (d=0;d<5;d++){
    fd = open("/dev/${devices[$d]}", O_RDONLY); //open /dev/sdX
    lseek(fd, offset, SEEK_SET); //seek to offset, starting from beginning of device
    read(fd, diskBuf[d], 1);
    printf("device %s reads 0x%02x", ${devices[$d]}, diskBuf[d]);
    close(fd);
}

calcParity = diskBuf[0] ^ diskBuf[1] ^ diskBuf[2] ^ diskBuf[3];
expectedParity = diskBuf[parIdx];

printf("Calculated parity: 0x%02x; parity read from sde: 0x%02x", calcParity, expectedParity);

 

Output:

device sda reads 0xAA

device sdb reads 0xAA

device sdc reads 0xAA

device sdd reads 0xAA

device sde reads 0xAA

Calculated parity: 0x00; parity read from sde: 0xAA

 

Offset 511 is an easy example; even parity (which unraid uses) should be 0x00. This one could be hand-waved away as 'oh, well maybe it's actually using odd parity or something'.

 

Thus:

device sda reads 0xF1

device sdb reads 0xF0

device sdc reads 0xF1

device sdd reads 0x70

device sde reads 0x71

Calculated parity: 0x80; parity read from sde: 0x71

 

I can't make any sense of this.

 

I don't think this has anything to do with the parity problem I reported in my other thread; this is consistent and has always returned the same values, the other is extremely flaky and inconsistent.

 

I poked around in the unraid kernel driver code and verified that (as far as I can tell), the XOR calculation for parity is done the same.

 

Could something in the unraid driver be interfering with my ability to directly read the parity device?  I figured the unraid driver should only come into play if accessing the /dev/mdN devices, rather than the block devices.

 

Has anyone else ever attempted something like this?

[Joe L.]

Disclaimer: very advanced topic. 99.9% of users will have no use for this. Intentionally posting code that doesn't compile lest someone try to do something they shouldn't.

Understood

devices = {sda, sdb, sdc, sdd, sde}

... snip ...

 

I can't make any sense of this.

You'll need to compare the first partition on each disk.  The MBR and area prior to the start of the first partition is not maintained by parity, but re-established as needed if a disk is ever replaced.  SO...  I've not compiled your source  (obviously ... compiler would choke on it),

but I think you want:

devices = {sda1, sdb1, sdc1, sdd1, sde1}

Has anyone else ever attempted something like this?

Yes, me... but I did it on a single byte on a single drive, to force bad parity in a test, and found the byte I changed had no effect. (it was before the start of the first partition)  Was confusing at first.  I never wrote a "C" utility to verify parity, but I have written a few hundred thousand lines over the years...    your utility is very interesting, as it might be a great tool to verify which disk is "inconsistently" causing errors.

 

As I said,I think you need the same offset relative from the start of partition 1 for it to make sense, not relative from the start of the physical disk.

 

The "md" devices are connected to the first partition of each respective disk.  It is what allows GPT partitions, and partitions starting on sector63, and starting on sector 64 to all work together.

 

Tom has said at one time long ago (before he added support for GPT partitions) that he might have to calculate parity on the entire drive... but I doubt that is necessary now that GPT partitions are in place.

 

My understanding is that the first byte of /dev/md1 (or /dev/sdX1), /dev/md2, etc xored with the first byte of the first partition of the parity disk will result in even parity.

 

Joe L.

[nick5429]

You'll need to compare the first partition on each disk.  The MBR and area prior to the start of the first partition is not maintained by parity, but re-established as needed if a disk is ever replaced.  SO...  I've not compiled your source  (obviously ... compiler would choke on it),

but I think you want:

devices = {sda1, sdb1, sdc1, sdd1, sde1}

 

Both important points: a) I was trying to compare areas of the disk that aren't actually protected for my testing and b) using the partition files fixes the sector 63/64 alignment issue.

 

My little program works nicely now! Thanks Joe.  PM me if you'd like a copy, I'm disinclined to post it here; potential for misuse is high, especially if/once tweaked to update parity.

 

I do have one real parity error on my drive from running a correcting parity update on my array after an unclean shutdown (knowing that I'd likely get 1-2 real parity errors introduced due to the flaky drive/controller/whatever I have going on, but it was better than the ~10-15 from the unclean shutdown)

 

syslog reports the parity mismatch at 2740517552; its offset per my program is 2740517555 from the beginning of the sdX1 partitions (using "1 block" = "512 bytes").  One single bit is flipped.

# ./parityCheck 2740517554

Calculated parity matches read parity

# ./parityCheck 2740517555

Printing block Parity sum across all devices including parity; should be 0 in hex:

0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000

0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000

0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000

0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000

0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000

0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000

0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000

0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000

0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000

0000000000000000000000000000000000000000000000000000000000000000000000000000000000001000000000000000

00000000000000000000000000

 

PARITY MISMATCH

 

It's pretty cool to be able to see/verify that

...and I might tweak it to fix the error just for that block on my parity disk if my 'switch the controllers' test doesn't work this evening.  Otherwise, there'll be no way to correctly rebuild the array when I swap the disk out (thus my feature request, as most users shouldn't have to / wouldn't be able to do something like this if presented with a problem like mine)