January 3Jan 3 I'd like to propose adding the disk type: "C" Checksum. Since unraid arrays vary in disk count (2 to 28 disks) it means a per-disk overhead would be wasteful. By tying checksum storage to array capacity we should scale better and allow us to compare against P/Q parity disks to create a quorum. Hashing MechanismsCRC64 (~10GB/s): Fast, sufficient for most bit rot scenarios and ~8 bytes/stripeBLAKE3 (~2GB/s): Faster than SHA256 but needs ~32 bytes/stripeSHA256 (~500MB/s): Thorough work horse but slow and ~32 bytes/stripeTheoretical example:If we take a 100TB array and assume a 4Kb page size this gives us 109,951,162,777,600 bytes or about 26.8 billion individual stripes. The checksum space required would be number of stripes * bytes per stripe or in our example 214,748,364,800 bytes (195~gb)Additional metadata would be required such as stripe index, metadata checksums, allocation tables, alignment/padding, etc. for all intents and purposes we can just throw an overhead factor of 2x. In our example a 100TB array would require a checksum disk equal to or larger than 390gbChecksums could be verified during every r/w operation if md_write_method is on. If a mismatch is detected:Log the corruption event via notificationIf recoverable via P/Q: automatically reconstruct to repair the stripeIf not recoverable: do something such as indicate the potential files corrupted for manual interventionCould also schedule the verification as a background task that runs during off-peak hours similar to mover or parity checks via tuning:Sequentially reads all array dataVerifies every stripe's checksumDetects AND repairs corruption proactivelyOptional: user-configurable frequency (weekly, monthly, etc.)
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.