-
5 Persistent Parity Errors on Same Sectors Two Weeks Apart
I think I've found the cause. 3519069768–3519069800 is exactly where a 4 TB drive's backup GPT lands if the motherboard firmware truncates the sector number to 32 bits. At boot it "restores" the backup partition table there, on drives attached to certain controllers (ASMedia/Marvell). A correcting check fixes parity, and the next boot writes it back. Details, a read-only check and a fix: https://forums.unraid.net/topic/201229-same-5-sync-errors-after-every-reboot-explained-the-bios-rewrites-a-backup-gpt-at-a-32-bit-wrapped-sector-asmedia-marvell/
-
Parity-Check - P corrected, sector= [...] - same 5 sectors
Re: the firmware update question. It won't help, because the write comes from the motherboard firmware at boot, not the controller. 2743151176–2743151208 is exactly where an 8 TB drive's backup GPT lands with the sector number truncated to 32 bits. Explanation and fix: https://forums.unraid.net/topic/201229-same-5-sync-errors-after-every-reboot-explained-the-bios-rewrites-a-backup-gpt-at-a-32-bit-wrapped-sector-asmedia-marvell/
-
-
"Same 5 sync errors after every reboot" explained: the BIOS rewrites a backup GPT at a 32-bit-wrapped sector (ASMedia / Marvell)
To be upfront Claude did the heavy lifting here, I just guided it along. TL;DR: if you get the same handful of parity sync errors after every reboot, at sectors like 3519069768–3519069800 (4 TB drives) or 2743151176–2743151208 (8 TB drives), on drives attached to an ASMedia (ASM1064/1164/1166) or Marvell (SAS2LP) controller, the cause is very likely the motherboard firmware, not your drives and not Unraid. At boot it "repairs" each drive's backup GPT partition table, but it computes the sector number with only 32 bits. So it writes 16.5 KiB of partition-table data into the middle of the disk, about 1.6 TB in on mine. A correcting check fixes parity, and the next boot writes it back. What I sawUnraid 7.3.4, dual parity, six 6 TB drives. Board: MSI PRO B550M-VC WIFI, whose SATA ports 5–8 are an onboard ASM1064. UEFI boot, CSM off. Every check after a reboot found the same 5 errors at sectors 3131110472, …480, …488, …496 and …504. A correcting check "fixed" them, and the next reboot brought them back byte-for-byte identical. Drives, cables, RAM and SMART were all clean, and no file had been written to that spot. What is actually at that spotReading the raw sectors with dd showed a GPT header ("EFI PART") and a partition entry array. The header's own MyLBA field says it belongs at the drive's real last sector. It was physically somewhere else: offset bytes field 0x00 45 46 49 20 50 41 52 54 signature "EFI PART" 0x18 af f4 a0 ba 02 00 00 00 MyLBA = 11,721,045,167 (real last sector of a 6 TB drive) 0x20 01 00 00 00 00 00 00 00 AlternateLBA = 1 found at LBA 3,131,110,575 = 11,721,045,167 - 2 x 2^32Only the drives on the ASMedia ports had this. The four drives on chipset ports didn't. Linux reads and writes those sectors correctly through the same controller, so the truncation is in the firmware's boot-time driver. Why it is exactly 5 errors (or 2)The firmware writes 33 sectors (the 32-sector entry array plus the header), ending at the wrapped sector. Unraid's partition starts at sector 64 and compares parity in 4 KiB pages (8 sectors). A 33-sector run that starts on sector 7 of a page always spans 5 pages. The middle 3 pages are almost all zeros (empty partition entries). If the correct parity there is also zero, you only see 2 errors, the first and last page. That matches the 2-error reports in threads 138669 (3 TB drives) and 151698. On a parity drive you get P or Q errors. On a data drive the data itself changed, so you get "PQ incorrect". The firmware only writes when the spot doesn't already hold a valid header. That's why a check straight after a correcting check finds 0, and why the errors only come back after a reboot. In thread 151698, Unraid runs as a Proxmox VM with an ASM1166 passed through, and the errors came back only when the physical host rebooted, never when the Unraid VM restarted. That fits firmware that runs only when the physical machine boots. Predicted sectors by drive sizeW = (total sectors − 1) mod 232. Unraid reports pages ⌊(W−96)/8⌋·8 through ⌊(W−64)/8⌋·8. Drives of 2 TB or less have fewer than 232 sectors and aren't affected. Drive Sync-error sectors Seen in threads 3 TB 1565565768 – 1565565800 77770, 138669 4 TB 3519069768 – 3519069800 56262, 138669, 151698, 197495 6 TB 3131110472 – 3131110504 this one 8 TB 2743151176 – 2743151208 130894 10 TB 2353004440 – 2353004472 12 TB 1962934168 – 1962934200 14 TB 1574961048 – 1574961080 16 TB 1186987928 – 1186987960 18 TB 796917656 – 796917688 20 TB 408944536 – 408944568 These use the standard sector counts for each size and 512-byte logical sectors (512n and 512e drives, which is nearly every hard drive). Check yours with blockdev --getss and --getsize64. A 4Kn drive (4096-byte logical sectors) only wraps above 16 TiB, at different sectors. The loop and the script below handle both. Check your own drives (read-only)for d in /dev/sd[a-z]; do ls=$(blockdev --getss "$d"); n=$(( $(blockdev --getsize64 "$d") / ls )) [ "$n" -gt 4294967296 ] || continue w=$(( (n - 1) % 4294967296 )) sig=$(dd if="$d" bs="$ls" skip="$w" count=1 2>/dev/null | head -c 8 | tr -d '\0') my=$(dd if="$d" bs="$ls" skip="$w" count=1 2>/dev/null | od -An -tu8 -j24 -N8 | tr -d ' ') if [ "$sig" = "EFI PART" ] && [ "$my" = "$((n - 1))" ]; then echo "$d: AFFECTED - firmware's backup GPT header at wrapped sector $w" else echo "$d: nothing at wrapped sector $w" fi doneA drive counts as AFFECTED only if the sector holds a GPT header whose MyLBA field is that drive's own last sector. That's the firmware's fingerprint, so a match can't be a coincidence. A drive on an affected controller shows it from its first boot on that controller, until a correcting check or rebuild overwrites the spot. The loop uses each drive's logical sector size. What fixed it for meParity drives off the affected controller. A parity drive can't be protected: its content at that spot is computed from the data drives, so every boot overwrites it again. I swapped two SATA cables so both parity drives are on chipset ports and two data drives are on the ASMedia ports. Unraid tracks drives by serial, so nothing needed reassigning. Fixed parity for those pages once (a correcting check does this). After that, a reboot left 0 errors for the first time. Protected the spot on the data drives still on the ASMedia ports. Today the firmware's write lands in free space, but if a file were ever stored there, the next boot would overwrite 16.5 KiB of it. I reserved exactly those 5 XFS blocks with a small immutable placeholder file: preallocate, punch out everything except the spot, collapse, verify with xfs_bmap -v, then chattr +i. The script is at the end of this post. The cleaner hardware fix is an LSI HBA in IT mode. The user in thread 130894 ran two LSI 9217-8i cards for years without this, and it started after switching to ASMedia cards. Things that did not help other people: an ASM1166 firmware update (thread 138669; the bug isn't in the controller's firmware), and moving drives between ports on the same controller (thread 151698). On my board CSM was already off, so it isn't the legacy option ROM either. If a data drive already has a file thereOn a data drive that has been on one of these controllers, the 16.5 KiB at that spot may belong to a file that has already been overwritten. Before correcting, check which file owns it, using the start and end sector numbers from your error log: xfs_io -r -c "fsmap -d START END" /mnt/diskN. A correcting check makes parity match the drive as it is now, so it won't bring a damaged file back. Restore that file from backup. The guard scriptIt's in the spoiler below (about 192 lines, bash, run as root, with usage steps at the top). Read it before you run it. bash gpt-guard.sh check is read-only. For every parity and data disk it shows the controller, the wrapped spot (worked out from the drive's size, logical sector size and its own GPT), and whether the firmware's header is really there: the "EFI PART" signature, a valid header CRC, and MyLBA equal to the drive's own last sector. For data disks it also shows what owns those blocks and what apply would do. It says plainly when a parity disk is being hit. check N ... limits it to the data disks you name (N is the disk number, so disk5 is 5). bash gpt-guard.sh apply N ... guards the data disks you name. It only acts on a disk with proof: the firmware's header at the wrapped spot. FORCE=1 overrides that, for example right after a rebuild wiped the header. It first checks that xfs_io has every command it uses and runs a dry run on a throwaway XFS in RAM (about 65 MB). The array must be started. It refuses a spot a file already owns, and removes its own file if any check fails. Afterwards it confirms, with xfs_bmap -p and XFS's own block-owner map, that the guard owns exactly those blocks as one unwritten extent, so nothing was written to the spot. Data disks only, XFS only. Tested on Unraid 7.3.4 (xfsprogs 7.2) with unencrypted XFS and 512e drives. check handles encrypted disks and 4Kn drives but neither is tested, so apply refuses 4Kn drives unless FORCE=1. Earlier threads with the same sector numbers: 56262, 77770, 130894, 138669, 151698, 197495. gpt-guard.sh
amitchell83
Members
-
Joined
-
Last visited