Swirl3208

Should we install the plugin realtek plugin if the device is integrated into the motherboard? For reference I have the RTL8125.

If the parity drive was using btrfs or zfs, wouldn’t each block have a checksum (according to this btrfs document on checksums) So unraid can know if this particular block on the parity disk is corrupted as well. If the checksum doesn’t match on the same block for disk 1 and parity drive, then it is definitely not recoverable (assuming we have 1 parity). But if the checksum is good on the parity disk + disk 2 - disk 7 for that block then we can safely reconstruct the block on disk 1. This is where I disagree. Currently as of today, unraid cannot recover a file. You are right that today, unraid can only reconstruct an entire disk. However I’m talking more about the possibility of implementing a recovery mechanism that utilizes the checksum (to validate the data is still healthy) and parity information (to rebuild the block) in the future so we can recover a small portion of the disk instead of the entire thing.

As of today, unraid’s parity mechanism doesn’t have this capability. But could it be implemented in the future like how I outlined above? Unraid can see which blocks are corrupt for a btrfs / zfs file system, then get an XOR of the bits on the parity + other disks to fix the corrupted blocks on the original drive. If a corrupted file was written to the drive then yes, reconstructing would result in a corrupted file. I’m assuming the emulated disk holds a healthy file because bit rot / bit flip doesn’t update the parity. Bit rot / flip is a physical change on the disk and it happens outside of unraid’s control. The parity only gets updated when unraid is writing to the disk (which isn’t the case with bit rot / bit flip). You keep mentioning that this file doesn’t exist on any other drive.. I get that. The parity information can be used to reconstruct the bits where the file lives. I think we are on different pages on how parity works / is used for. I’m curious what do you think the parity is there for?

I guess I'm having a hard time trying to explain what I have in my head as well. I'll try in 2 different ways, a high level thought and a lower level appraoch. For the high level thought Think about it this way - Today, if 1 drive dies, unraid can emulate it while it is offline and we can still grab files from the emulated disk. Then to recover we should take the failed disk out and replace it with a new one. Unraid will start reconstructing the entire disk. After this you have a completely normal array again. This process is defined in the Normal Replacement or Rebuilding a drive onto itself guide from the unraid support pages. However reconstructing the disk does have a risk of another drive failing. When a corrupted file is detected on disk 1, unraid could easily "emulate" the corrupted file from parity + disk 2 - disk 7, then rewrite the file to disk 1 as a correction mechanism (actually in unraids case it doesn't even need to write back to the same disk, it only needs to be written back to the same share). Does that make sense? The benefit I see from this is that you don't have to rebuild an entire disk, which reduces the risk of another drive failing in the process. On a very low level approach: The Parity-Protected Array document states: "To rebuild the data on the newly replaced disk, we use the same method as before, but instead of solving for the parity bit, we solve for the missing bit." Unraid knows on disk 1 that my_file.txt is corrrupted. Unraid also knows that my_file.txt is on bit (or "column") 123 to bit 1000. Unraid could theoretically go into all the other drives and recalculate the bits 123 - 1000 on disk 1. That's a very true point. I can probably afford to replicate the important stuff like documents. For this topic I'm trying to see if we recover as early in the process as possible.

@strike Let's say that we have 1 parity + 8 disks in our array. Scenario 1 with xfs + file integrity plugin: If unraid found that the file was corrupted on disk 1. Why can't unraid go to the exact spot the file starts and ends in the parity drive and disk 2 - disk 8 and do and XOR operation on all the bits in that range to recover the data on drive 1? Scenario 2 with btrfs or zfs: Alternatively if our drives are formatted with btrfs or zfs, why can't unraid go to the exact same block in the parity and disk 2 - disk 8 and do an XOR to recover the corrupted block on disk 1? (Although since zfs has dynamic block size we might need to do scenario 1) I know this isn't how unraid works today but is this something that could possibly be implemented in the future? @JorgeB Currently with today's tools, how do we recover if the file integrity plugin or the filesystem detects rot? One solution I can think of is if we detect a corrupted file on disk 1. We could proceed to Rebuild a drive onto itself. This method will use the parity information from the parity drive + disk 2 - disk 8 to rebuild the entire drive. My question asking why can't we just rebuild a small section of the hard drive that we know is corrupted instead of doing the entire hard drive

This is a good point, I forgot about this scenario, which is definitely more likely than bit rot. I was too focused on bit rot in my last reply. I'm not quite understanding this part, if the parity drive can recover an entire drive, then it should be able to recover data for 1 block within the drive. From what I understand the parity drive is just an XOR operation of all the data drives. When 1 entire disk fails, unraid will rebuild the disk by taking an XOR of the parity drive + other drives. When a block on a disk fails, why wouldn't we be able to take an XOR of the block on the parity drive + other drives to rebuild that block? The parity calculation should be the same whether its unraid or zfs. The difference is that unraid stores all the parity data on 1 drive, where zfs stripes the data across a row, and stores the parity in different drives. References for zfs parity "Data and parity is striped across all disks within a raidz group": https://openzfs.github.io/openzfs-docs/Basic Concepts/RAIDZ.html View the table under the "RAIDZ (RAID5)" section: https://www.raidz-calculator.com/raidz-types-reference.aspx

@Duckers I believe the tasks run one after another. They used to be called groups as well in the past if that helps you understand a bit better. Lets say in Task 1 you put disk 1 and disk 2. In Task 2 you put disk 3 and disk 4. The plugin will start calculating the hash for disk 1 and disk 2 in parallel. When those 2 disks are finished running, the plugin will calculate the hash for disk 3 and disk 4 in parallel.

If we are using ZFS or BTRFS in the array, and bit rot is detected (assuming only on 1 disk), does unraid automatically recover and rebuild the corrupted block using the parity information? If not, then how do we recover when bit rot is detected on 1 drive? On the other hand if we are using xfs with the file integrity plugin, how do we recover from bit rot detection?

Theoretically, for an array with 1 parity drive, we could recover from bit rot if only 1 drive has corrupted data for that block. Using zfs checksum mechanism, If we know the block on disk1 is corrupted then we can use the parity information from the other drives to rebuild the block. This idea is similar to how zfs currently self heals like you said. Yes everyone should have a backup, but not everyone can afford the setup for a backup. The backup should be a last resort option. The reason why we have a parity drive and these other checks is to recover faster without going to the backup. Otherwise what's the point of the parity drive? Currently with xfs if there's a parity check failure, we can't tell which drive is failing the parity check, so we don't know which drive to fix with the parity data.

Even though bit rot or a bit flip is very rare, I would like to protect against it if possible. I saw space invader one's video about 6.12 and saw that he added a ZFS formatted drive into the Unraid Array (not a zPool). As I understand we now have 3 ways to detect bit-rot: xfs + file integrity, btrfs, and zfs. When bit rot is detected do any of these 3 options have mechanism to recover from bitrot? Like could Unraid use the parity drive to recover the data on that block?