unRAID Server Release 4.7 "final" Available


limetech

Recommended Posts

So with the line:

 

Aug 28 07:14:21 Tower kernel: ata3.00: HPA detected: current 1465147055, native 1465149168 (Minor Issues)

 

in my syslog, I would use:

 

hdparm -N p1465149168 /dev/sdb

 

right?

 

Yes, but beware, and follow the instructions from the other thread... Only 1 disk at a time, and rebuild it after removing the hpa

 

You should check the thread to see if you need to unmount etc. I removed the disk and removed the hpa in another machine, and this made unraid see it as a completely new disk.

Link to comment
  • Replies 414
  • Created
  • Last Reply

Top Posters In This Topic

Top Posters In This Topic

Posted Images

I booted back into 4.6 and ran the needed hdparm lines and then rebooted. On reboot there is nothing in the syslog about HPA, so I'm feeling good about that. Disk2 is in the process of rebuilding now. Once it's done I'll do a full parity check and then try to reinstall 4.7 and from there a 5.0rc. Then I can start a preclear on this new 3tb hdd and if all goes well replace my current parity drive with it. Thanks for all the help guys.

Link to comment

I booted back into 4.6 and ran the needed hdparm lines and then rebooted. On reboot there is nothing in the syslog about HPA, so I'm feeling good about that. Disk2 is in the process of rebuilding now. Once it's done I'll do a full parity check and then try to reinstall 4.7 and from there a 5.0rc. Then I can start a preclear on this new 3tb hdd and if all goes well replace my current parity drive with it. Thanks for all the help guys.

 

I've seen this before, but if you trust Unraid (as you should) a parity check right after a rebuild should not be necessary...

 

Although while writing this I remembered there is a bug in 4.7 which could allow bad parity while rebuilding and writing to the array at the same time (and the same sector I think).

 

So just do the extra parity check just to be sure.

 

Link to comment

A parity check is required to verify the rebuild.

 

If I'm rebuilding a drive using parity, and the parity is not correct, the rebuild would not be correct, but the parity check afterwards would be...

 

If I'm rebuild a drive using correct parity, but the new drive is defective (but still manages to be rebuild), and I perform a correcting parity check, I'm correcting parity with the bad drive data and losing data.

 

Only thing which might be sensible to do is do a no-correct parity check after a rebuild to verify the new drive. I would not say it is 'required'.

Link to comment

A non correcting check is a good idea, maybe not required. If parity is wrong then there is corruption you can't help. If parity is right and there is corruption during the rebuild, which was a bug in a 4.x release, not sure what version, then a non correcting check will let you know.

 

If this does happen and is not fixed, a future parity check will make parity wrong, especially if the server was not cleanly shut down, the parity check on start up will cause parity to be updated with corrupt data.

 

So not required, but a good idea to ensure your rebuild went without corruption.

Link to comment

It is the only way to verify that what was written to the new disk can be read. ...

That's an important point. But not just for new disks.

 

It also applies at the block level, for any write to the array. I believe there is merit in being able to perform a selective/targeted parity check. That is, parity checking only those sectors/blocks/stripes "recently" written. I'll leave it to you to define "recently".

 

Link to comment

It is the only way to verify that what was written to the new disk can be read. ...

That's an important point. But not just for new disks.

 

It also applies at the block level, for any write to the array. I believe there is merit in being able to perform a selective/targeted parity check. That is, parity checking only those sectors/blocks/stripes "recently" written. I'll leave it to you to define "recently".

Well, once parity is established a "read" is performed immediately prior to a write of a sector in order to calculate the needed change to parity.  Therefore, there is a better chance the sectors involved are readable sectors.  If that read fails, the "md" driver re-constructs it from parity and the other data disks, writes it (hopefully allowing SMART re-allocation to occur) and then continues with the write of the block being written. 

 

On new data disks, without parity, you'll not know of an un-readable sector until a read is attempted.

 

Joe L.

Link to comment

It is the only way to verify that what was written to the new disk can be read. ...

That's an important point. But not just for new disks.

 

It also applies at the block level, ...

Well, once parity is established a "read" is performed immediately prior to a write of a sector in order to calculate the needed change to parity.  Therefore, there is a better chance the sectors involved are readable sectors.

Yes, I am aware of all of that. I should have communicated the crux of my point more precisely. In the context of this discussion, there are two notions of a sector. One is the location (LBA / "lot # on a tax map"). The other is the current contents (raw bit-stream (PRML-encoded) + ECC / actual house currently occupying the lot (construction/build-quality, etc)).

 

Iow, even though the lot is level, solid, well-drained etc., the builder might be sub-par [even, if only on the day he built this house]. As you pointed out, unRAID is less susceptible in one regard. Because it reads the parity block (just) prior to re-writing it, the heads are already on track, so alignment issues are reduced. But there is still the possibility of a miscue/botch in the write process.

 

==> It is for this reason that read-after-write gets so much attention when (super-)high-data-integrity is discussed.

 

Back to unRAID ... while the parity block re-write has the "benefit" of a reduced likelihood of mis-alignment, that is not the case for the newly-written data block. A targeted parity check will give added assurance for both. Actually, just a read of both sectors might suffice, if a re-construct were implicit upon either failing. Murphy-alert!: what about read-after-write following a re-construct? :)

 

So, what I am suggesting, is a batched post-facto selective read-after-write. Either of us could write it in a few hours for unRAID (but only one of us might be motivated :)).

 

I don't use unRAID. I prefer SnapRAID for my needs; I've added this feature for my use (peace of mind) and suggested it to SnapRAID's author [link].

 

 

 

Link to comment

It is the only way to verify that what was written to the new disk can be read. ...

That's an important point. But not just for new disks.

 

It also applies at the block level, ...

Well, once parity is established a "read" is performed immediately prior to a write of a sector in order to calculate the needed change to parity.  Therefore, there is a better chance the sectors involved are readable sectors.

Yes, I am aware of all of that. I should have communicated the crux of my point more precisely. In the context of this discussion, there are two notions of a sector. One is the location (LBA / "lot # on a tax map"). The other is the current contents (raw bit-stream (PRML-encoded) + ECC / actual house currently occupying the lot (construction/build-quality, etc)).

 

Iow, even though the lot is level, solid, well-drained etc., the builder might be sub-par [even, if only on the day he built this house]. As you pointed out, unRAID is less susceptible in one regard. Because it reads the parity block (just) prior to re-writing it, the heads are already on track, so alignment issues are reduced. But there is still the possibility of a miscue/botch in the write process.

 

==> It is for this reason that read-after-write gets so much attention when (super-)high-data-integrity is discussed.

 

Back to unRAID ... while the parity block re-write has the "benefit" of a reduced likelihood of mis-alignment, that is not the case for the newly-written data block.

Incorrect. Both parity and data blocks are read before being written in order to update parity.

A targeted parity check will give added assurance for both. Actually, just a read of both sectors might suffice, if a re-construct were implicit upon either failing. Murphy-alert!: what about read-after-write following a re-construct? :)

 

So, what I am suggesting, is a batched post-facto selective read-after-write. Either of us could write it in a few hours for unRAID (but only one of us might be motivated :)).

 

I don't use unRAID. I prefer SnapRAID for my needs; I've added this feature for my use (peace of mind) and suggested it to SnapRAID's author [link].

 

A parity check after parity build or data rebuild is required to verify what was written. Normal data writes occur after the sector is read, additionally the parity protection will correct an unreadable data sector. For both of these reasons a read after a normal data write is not necessary.

Link to comment

... while the parity block re-write has the "benefit" of a reduced likelihood of mis-alignment, that is not the case for the newly-written data block.

Incorrect. Both parity and data blocks are read before being written in order to update parity.

"You are correct, sir." Thank you.

But, remember, this only partially eliminates susceptibility to (lateral) misalignment; and there are numerous additional ways for a write to go wrong.

A parity check after parity build or data rebuild is required to verify what was written. Normal data writes occur after the sector is read, additionally the parity protection will correct an unreadable data sector. For both of these reasons a read after a normal data write is not necessary.

Trust, but verify.  Right?

And the sooner you verify, the narrower the window of vulnerability on the worthiness of your trust. (Unless they are used/read during day-to-day operation) All of these normal data block writes will never have had their readability confirmed, not to mention the correctness of their contents verified, until a (full) parity check is performed. At that time, I agree, either of those flaws will be detected and corrected.

 

But, how often does a typical user perform a parity check? Yeah, there might be one or two fanatics that do it nightly, but I would guess typical is monthly, with a sizable minority doing it weekly. This is the window of vulnerability. If you have a drive failure AND you have any of those untested/unverified newly-written data blocks (on a non-failed drive) turn out to be bad, your replacement/rebuild of the failed drive will be less than perfect.

 

Since it is normal, and recommended, to do a (full) parity check directly following a (full) build/rebuild, why would it not be prudent to do a (partial/targeted) parity check directly following a cache-to-array "mover session" (ie, a partial rebuild)? Granted, a proper partial parity check will require driver mods (but they're not difficult--couple of new [fs/io]ctl's). But, just reading those newly-written blocks will get 90+% of the effect, which is to eliminate that (broad) window of vulnerability.

 

Just trying to help you keep your data safe® ...

 

 

 

Link to comment

A true battle of the wits! It is a great read and has given me insight to how unraid works behind the scene.

 

What unclem wrote makes sense, although I don't know unraid well enough to know how necessary it is. But the logic seems sound and I think would be a great addition. And considering most users I think do not add a great deal to their array on a daily basis, a targeted check would be a nice alternative, or should I say addition, to the monthly parity check.

Link to comment

If you don't trust it when you write a sector to a hdd, and the hdd returns ok, and only trust it once you've read it again....

 

that would me you can't trust it when you've read it again, as it might have gone bad, and this would mean the best thing would be to do a continuous parity check...

 

You have to trust a disk and it's return messages at some point, and I trust the disk when I've written to it and it returns an ok. If I don't trust that, I cannot trust hdd's at all.

 

I realize a sector can go bad while not reading/writing it at all.

 

But that can occur on the parity disk as well as any other data disk, which would mean when this happens (a random sector goes bad) you cannot even tell if you want to trust parity or the data disks unless you trust the hdd which should give a bad sector/crc error. This would mean you would have to analyse the specific error and either rebuild the data disk with the error or the parity disk.

 

This whole example is not directly related to the rebuild and it's proceeding actions, this can happen at any time. So for me, rebuilding without a write error is reliable enough.

 

I see the point of regular parity checks, but as these run for 20+ hours in my server (4tb parity, 3tb data disks) and stress my system more than normal usage, I do not want to run it too often. Ofcourse we'll be getting into a discussion if it's better to run/stress a system 24/7 or have it sleep most of the time, at system level or component level.

 

 

 

 

Link to comment

If you don't trust it when you write a sector to a hdd, and the hdd returns ok, and only trust it once you've read it again....

You're missing a very important distinction here. What is key is not reading it again, but reading it (the newly-written sector) the first time.

 

In other words: "never being good in the first place" is quite different from "going bad (over time)".

 

Also, you are putting too much (blind :)) faith in the (ATA) write command returning OK. In practice, the only time a write command will return ERR is if the spare sectors are fully depleted (or if faulty memory or controller has sent a mis-constructed ATA write command [ie, LBA out-of-range]).

 

(In contrast, a readOK is reliable, because of the ECC, etc.)

I see the point of regular parity checks, but as these run for 20+ hours in my server (4tb parity, 3tb data disks) and stress my system more than normal usage, I do not want to run it too often.

Exactly!!! And one shouldn't proofread their entire thesis every time they revise only a single paragraph. But they better proofread the changed paragraph (lest they embarrass themself  to their advisor(s)).

 

--UhClem "We're all bozos on this bus."

 

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.