How does UNRAID "emulate" writes to a disabled device?

[XanderLort]

Recently had a drive fail and was reading on the troubleshooting procedures, which talk about potential writes to the emulated drive? Where do those writes go? Are they lost upon a reboot? Are they preserved in the parity disk? If they are preserved in the parity disk, would the writes survive someone doing a "trust my array" operation?

 

Referenced troubleshooting steps taken from here: https://wiki.unraid.net/Troubleshooting#Re-enable_the_drive

[BRiT]

It updates the Parity Drive(s) as if the write(s) to the data drive happened. Then when you replace the faulty drive and the system is rebuilding the drive, it reads from all the other data drives and parity drives to calculate what the data should be for that drive.

[XanderLort]

1 minute ago, BRiT said:

It updates the Parity Drive(s) as if the write(s) to the data drive happened. Then when you replace the faulty drive and the system is rebuilding the drive, it reads from all the other data drives and parity drives to calculate what the data should be for that drive.

So, if I did a "Trust My Array" operation, but did not do a parity-sync, the writes should be preserved for when I replace and rebuild that drive?

[JonathanM]

2 minutes ago, XanderLort said:

So, if I did a "Trust My Array" operation, but did not do a parity-sync, the writes should be preserved for when I replace and rebuild that drive?

NO. Definitely not.

 

Trust my array implies that you want to rebuild parity from what is on the data drives, which would NOT include any writes that occurred after the drive was disabled.

 

Rebuilding will write the contents of the emulated drive to the physical disk.

[BRiT]

39 minutes ago, XanderLort said:

So, if I did a "Trust My Array" operation, but did not do a parity-sync, the writes should be preserved for when I replace and rebuild that drive?

No. Absolutely not. Wrong thing to do. 

 

 

[XanderLort]

55 minutes ago, jonathanm said:

NO. Definitely not.

 

Trust my array implies that you want to rebuild parity from what is on the data drives, which would NOT include any writes that occurred after the drive was disabled.

 

Rebuilding will write the contents of the emulated drive to the physical disk.

A "Trust My Array" operation does not re-calculate the parity though, as when it comes back up you check the "Parity is already valid" box. By that logic, as long as you don't re-sync the parity you should be able to re-build the lost drive with the emulated writes.

 

Addendum: I understand it is bad practice and a generally terrible idea, I am just trying to make sure there isn't an intermediate product being stored that would remove the emulated write data from the parity drive.

[XanderLort]

27 minutes ago, BRiT said:

No. Absolutely not. Wrong thing to do.

(See previous response - forgot to multiquote)

[JonathanM]

4 minutes ago, XanderLort said:

A "Trust My Array" operation does not re-calculate the parity though, as when it comes back up you check the "Parity is already valid" box. By that logic, as long as you don't re-sync the parity you should be able to re-build the lost drive with the emulated writes.

But parity is NOT valid at that point, and checking that parity is valid would put the array in an inconsistent state. The data drive takes precedence if it is readable, so when you accessed the newly enabled (not rebuilt) drive it would show that all writes after the drive was disabled were lost. If you did a correcting parity check, it would show errors at all sectors where there were writes, and parity would be updated to agree with the outdated data drive.

[XanderLort]

38 minutes ago, jonathanm said:

But parity is NOT valid at that point, and checking that parity is valid would put the array in an inconsistent state. The data drive takes precedence if it is readable, so when you accessed the newly enabled (not rebuilt) drive it would show that all writes after the drive was disabled were lost. If you did a correcting parity check, it would show errors at all sectors where there were writes, and parity would be updated to agree with the outdated data drive.

Agreed. But, as long as you don't do a parity update and don't do any more writes to the drive, you should be able to replace the problematic drive and re-build it with the contents of the parity, which include the emulated writes, at which point everything is correct and all data is available. So the sequence of events would be:

• Drive encounters a write error and gets disabled with emulation (non-parity drive).
• Writes occur to the disabled drive, actually get written to parity-only. At this point, the parity is invalid as compared to the array in the written sectors.
• Do a "Trust My Array", do not re-calculate the parity. Do not write anything else to drive.
• Replace the drive and re-build that drive.
• After re-build is complete, drive is restored with emulated writes and parity is in sync.

 

Or am I missing something?

 

(This is all for conceptual understanding only - I'm a software engineer, so I'm not being intentionally frustrating )

[JonathanM]

I'm obviously missing something.

What do you intend to accomplish with the trust my array procedure?

 

Your bullet point list is fine, as long as you remove the 3rd line entirely.

 

23 minutes ago, XanderLort said:

• Drive encounters a write error and gets disabled with emulation (non-parity drive).
• Writes occur to the disabled drive, actually get written to parity-only. At this point, the parity is invalid as compared to the array in the written sectors.
• Replace the drive and re-build that drive.
• After re-build is complete, drive is restored with emulated writes and parity is in sync.

 

[XanderLort]

4 minutes ago, jonathanm said:

I'm obviously missing something.

What do you intend to accomplish with the trust my array procedure?

 

Your bullet point list is fine, as long as you remove the 3rd line entirely.

I am not trying to accomplish anything with the "Trust My Array" step - only trying to gain a conceptual understanding of how it, the parity calculation process, and the drive rebuild process interact with each other. My point is that after you do a Trust My Array, the parity still contains the emulated write data until it is overwritten by either a parity update operation, or additional writes (to the same sectors). Rebuilding a drive with that data in the parity will include the emulated writes.

 

I think... it might be possible to test using an externally removable drive and removing it in the middle of a write.

[trurl]

The "normal" way Unraid updates parity is to only read parity and disk to be written and use that to calculate the change to parity that results from the data to be written. So if parity and data are out-of-sync before a write, the wrong change will be calculated.

 

 

[XanderLort]

3 minutes ago, trurl said:

The "normal" way Unraid updates parity is to only read parity and disk to be written and use that to calculate the change to parity that results from the data to be written. So if parity and data are out-of-sync before a write, the wrong change will be calculated.

 

 

Correct, which is why the emulated writes to the parity only will persist until they are specifically overwritten by writes to those sectors (or a parity update operation across the whole array).

[trurl]

Just now, XanderLort said:

Correct, which is why the emulated writes to the parity only will persist until they are specifically overwritten by writes to those sectors (or a parity update operation across the whole array).

After you trust the array, the disk is no longer emulated. But the physical disk is out-of-sync with parity. Parity has the emulated data that was written before, but the actual contents of the physical disk is what will be used to calculate the parity change when the out-of-sync sectors are written again. So an incorrect parity would get written.

[trurl]

So I guess you could say that as long as you don't write to the disk that needs to be rebuilt until after you rebuild it, it can still be rebuilt correctly. But what would be the point of not just proceeding with the rebuild? Just because you could rebuild later doesn't really buy you anything since the array is still unprotected until everything is back in sync. Might as well just leave it disabled and emulating.

[XanderLort]

1 minute ago, trurl said:

So I guess you could say that as long as you don't write to the disk that needs to be rebuilt until after you rebuild it, it can still be rebuilt correctly. But what would be the point of not just proceeding with the rebuild? Just because you could rebuild later doesn't really buy you anything since the array is still unprotected until everything is back in sync. Might as well just leave it disabled and emulating.

Agreed - this whole thing is a hypothetical situation where someone did not know there were emulated writes to the disk before doing a "trust my array" operation, but then wanted to recover those writes. Not sure if there is a mechanism for knowing whether there were emulated writes to the disk or not.

[JonathanM]

10 hours ago, XanderLort said:

Agreed - this whole thing is a hypothetical situation where someone did not know there were emulated writes to the disk before doing a "trust my array" operation, but then wanted to recover those writes. Not sure if there is a mechanism for knowing whether there were emulated writes to the disk or not.

Ahh, now it becomes clear. The likely outcome of your hypothetical would be a corrupted filesystem, with recovery uncertain, depending on how badly the filesystem was tramped on by the out of sync writes. Yes, those writes could be "recovered", but they would be out of sync with the current status.

 

What you are failing to take into account is that there are housekeeping writes done during disk mounting, so purposely not writing files in the hopes of keeping the parity changes intact would be torpedoed by the normal array operations.

 

Bottom line, "trust my array" is NOT something you want to be doing unless you are forced by some extra circumstance beyond a simple disk failure. It is risky, which is why it's a manual procedure not something with an automatic single button push.