Yes for the drive, not global... and we're 6 hours in pass 2, 1.49 tb progress and no errors...

if it makes it to 2tb without them.... I'll wait for the other 2tb to finish (not gonna jinx it) before I declare anything

OK, second pass finished with 0 errors.. so, let me recap and see if I got this right

 

I had this system running since early 6.0, like b10 IIRC, I built it with three drive and migrated an older system to it copying one drive at a time and then adding the freshly emptied drives to it. The only 'full' parity build had been when I started it and it had noting to really do (empty drives).

 

Since then I had just been adding to it, it was running the monthly parity checks with no errors, had never replaced the parity drive. I HAD replaced two data drives with rebuilds (upped them to bigger drives, no redballs) without issue.

 

Now, put in a new 4tb drive for parity for future expansion. This did do a full parity rebuild. That APPARENTLY (gonna use that word a lot here) when the trouble reared its head

 

- the rebuild finished ok, drive spindown was set to default

- I did a non-correcting check and that's when the 80 errors showed up (all consecutive sectors). Strangely enough it still said "Parity Was Valid"

- APPARENTLY the parity was mis-written during the rebuild, then when checked, it was wrong

- at this point my lack of knowledge kicked in (for some reason I though it would correct the data to match the parity on a correcting check.. duh)

- after several non-correcting checks I finally got sense into my head and ran it again with correct errors (drive spindown was still active)

- the correction check APPARENTLY corrected 80 errors.. but when I re-ran a second (non-correcting check) it found the same 80 again, yes "Parity was valid"

- turned spin down OFF for the parity drive,

- re-ran the correction check again and again it said corrected 80 errors

- ran the correction check again immediately.. and 0 errors

 

so....

 

Do I need to leave the parity drive spinning 'ever more? is there any sort of idea what's happening?

Is there any idea of it being a weird combo of mb/cpu/controller/drive? (Gigabyte F2A85XM-D3H, AMD A6-6400K (mb controller for the parity),WD40EZRX-00SPEB0)

If I turn spindown back to normal and do a correcting check, can I expect this same brouhaha? Then have to turn spindown off and redo the correcting check?

Any though of having the system turn OFF spindown when it's doing the correcting check?

Bueller?

Not sure what combination of components results in this anomaly, but as you've seen from the link I provided earlier, others have had the same issue.

 

In any event, all's well now.    Worst case is you may have to disable the parity drive spindown before doing parity checks.    I'd re-enable spindown for parity and run another check to see if the issue re-occurs.    If so, you simply need to disable parity drive spindown when you want to check parity.    I'd check again after each UnRAID update to see if it's been resolved.    Limetech is aware of this issue, but hasn't been able to replicate it on any of their systems, so it's a tricky one to diagnose.    It's likely some specific combination of hardware ... but nothing's "jumped out" from the few folks who have experienced the issue.

 

 

Not sure what combination of components results in this anomaly, but as you've seen from the link I provided earlier, others have had the same issue.

 

In any event, all's well now.    Worst case is you may have to disable the parity drive spindown before doing parity checks.    I'd re-enable spindown for parity and run another check to see if the issue re-occurs.    If so, you simply need to disable parity drive spindown when you want to check parity.    I'd check again after each UnRAID update to see if it's been resolved.    Limetech is aware of this issue, but hasn't been able to replicate it on any of their systems, so it's a tricky one to diagnose.    It's likely some specific combination of hardware ... but nothing's "jumped out" from the few folks who have experienced the issue.

That's why I wonder if the short term brute force solution is for a pastry chef to turn off spindown and restore it when done. But APPARENTLY the error happened when it was a full rebuild, not the check, but can't be fixed on a correcting check (properly rewritten) unless spin down is off.

 

Yeah this is a real weird one. About to do a drive upgrade on a data drive now, ye haw!

 

Oh geezer, parity check not pastry chef. Damn auto correct!

A possible scenario, just hypothesizing from the ivory tower -

 

* Under certain conditions, a parity check may result in a repeatable set of errors, probably spurious, but apparently involve inaccurate data returned at a specific and repeatable point in time.

 

* Disabling spin down for the parity drive appears to *fix* the errors.  This is obviously very strange as there is no possibility that a parity drive should be legitimately spun down during a parity check.

 

* Tom has indicated a desire to do away with spin groups, reason unknown, but would seem to indicate the possibility that a recent feature or a feature to be added may have tricky interactions, tricky to code around, if spin group functionality is preserved.

 

* Is it possible that the parity drive is assigned to a spin group with another drive, much smaller, that legitimately attempts to spin down while parity check is still running?  Perhaps there is a spin group 'tricky interaction' that attempts to erroneously spin down the parity drive too, and then even crazier, causes erroneous data returned during the spin down and then spin up of the parity drive.  The system of course would never expect that the parity drive could be unavailable at this time, so error checking may be relaxed.

 

* Why is it 5 errors in one case and 80 in another?  The time to spin a drive down then up can be surprisingly different, for different drive models.

 

* I note that the issue did not occur in this thread until a larger drive was introduced, if I understand correctly?  That introduces the possibility of a smaller drive being completed and available for early spin down.

 

* Disabling spin down for the parity drive avoids the spurious spin down, possibly explaining why that works.  Something that is testable then would be to re-enable the preferred spin down setting for the parity drive, then check the spin group settings, and make sure the parity drive is in its own unique spin group, so no possible spin group interaction.

 

* No ideas at all why wrong data gets through that long interval.  It should have resulted in timeout errors, or more.  But we do know that weird things *can* happen during that spin up or spin down interval, particularly to drives on the same channel or otherwise linked.

* Is it possible that the parity drive is assigned to a spin group with another drive, much smaller, that legitimately attempts to spin down while parity check is still running?  Perhaps there is a spin group 'tricky interaction' that attempts to erroneously spin down the parity drive too, and then even crazier, causes erroneous data returned during the spin down and then spin up of the parity drive.  The system of course would never expect that the parity drive could be unavailable at this time, so error checking may be relaxed.

In my circumstance, I have enable spin groups set to 'no'

* Why is it 5 errors in one case and 80 in another?  The time to spin a drive down then up can be surprisingly different, for different drive models.

and the size of the drive, 4tb in this case

* I note that the issue did not occur in this thread until a larger drive was introduced, if I understand correctly?  That introduces the possibility of a smaller drive being completed and available for early spin down.

Correct, but this was also the 1st time a full parity rebuild was done since the array was created. (the new larger drive was the parity) until this point the system had 3 and 2tb drives (with this issue NEVER happening), I noticed the failure happened sometime BEFORE 1.8tb had been completed. That's smaller than 2tb.

* Disabling spin down for the parity drive avoids the spurious spin down, possibly explaining why that works.  Something that is testable then would be to re-enable the preferred spin down setting for the parity drive, then check the spin group settings, and make sure the parity drive is in its own unique spin group, so no possible spin group interaction.

Again, no spin group defined. I think what I've said before makes more sense, when staring a parity rebuild or check (either style) disable spindown for the parity drive(s), then when completed, re-nable the previous setting.

* No ideas at all why wrong data gets through that long interval.  It should have resulted in timeout errors, or more.  But we do know that weird things *can* happen during that spin up or spin down interval, particularly to drives on the same channel or otherwise linked.

Computer... the home office if Weird'R'Us

A possible scenario, just hypothesizing from the ivory tower -

 

* Under certain conditions, a parity check may result in a repeatable set of errors, probably spurious, but apparently involve inaccurate data returned at a specific and repeatable point in time.

 

* Disabling spin down for the parity drive appears to *fix* the errors.  This is obviously very strange as there is no possibility that a parity drive should be legitimately spun down during a parity check.

 

* Tom has indicated a desire to do away with spin groups, reason unknown, but would seem to indicate the possibility that a recent feature or a feature to be added may have tricky interactions, tricky to code around, if spin group functionality is preserved.

 

* Is it possible that the parity drive is assigned to a spin group with another drive, much smaller, that legitimately attempts to spin down while parity check is still running?  Perhaps there is a spin group 'tricky interaction' that attempts to erroneously spin down the parity drive too, and then even crazier, causes erroneous data returned during the spin down and then spin up of the parity drive.  The system of course would never expect that the parity drive could be unavailable at this time, so error checking may be relaxed.

 

* Why is it 5 errors in one case and 80 in another?  The time to spin a drive down then up can be surprisingly different, for different drive models.

 

* I note that the issue did not occur in this thread until a larger drive was introduced, if I understand correctly?  That introduces the possibility of a smaller drive being completed and available for early spin down.

 

* Disabling spin down for the parity drive avoids the spurious spin down, possibly explaining why that works.  Something that is testable then would be to re-enable the preferred spin down setting for the parity drive, then check the spin group settings, and make sure the parity drive is in its own unique spin group, so no possible spin group interaction.

 

* No ideas at all why wrong data gets through that long interval.  It should have resulted in timeout errors, or more.  But we do know that weird things *can* happen during that spin up or spin down interval, particularly to drives on the same channel or otherwise linked.

 

Wow RobJ! You will win the sleuth of the century award if any of this turns out to be true! Then you can go on to deduce who shot Kennedy! I'm impressed with your ability to assemble and relate these seemingly unrelated topics into a believable story!!! Even if not true it still deserves a bravo!

Indeed a good set of conjectures ... however, as rick.p noted, not what's happening.    The errors aren't happening at a drive-transition point, and indeed not even some time after a drive transition point (which actually more likely what would be the case, since a drive has to be "uninvolved" in the check for the length of the spin-down timer before this would occur).  But since the errors occur before the smallest drives reach that stage (i.e. at 1.8TB with the smallest drives before 2TB) the sequence simply isn't what's happening.

 

Agree it's weird that the # of errors was 5 in the 4 other cases that reported this issue, but 80 in this case.  I still think whatever's happening is likely common for all cases, but there's certainly nothing that "jumps out" from any of the cases.  I know JonP is aware of this, and I presume is following the dialogue ... but he's also noted that LT has not been able to replicate this error, so it's going to be tricky to isolate.

 

It appears likely that the errors are not real ... likely just a "hiccup" in the data from the parity drive => but it's a very consistent "hiccup", which is really strange.    And it's REALLY weird that simply disabling spin-down on the parity drive resolves it.  Definitely one of life's little mysteries ...

 

Note for Rick.P => in the other thread I referenced, after disabling parity spin-down and running a good check, Justin re-enabled spin-down on the parity drive and than got a good (error-free) check.  Have you tried that after your good check to see if you have the same results?

 

Note for Rick.P => in the other thread I referenced, after disabling parity spin-down and running a good check, Justin re-enabled spin-down on the parity drive and than got a good (error-free) check.  Have you tried that after your good check to see if you have the same results?

 

Not yet, It's still 5 hours from finishing the rebuild of the 2tb drive I replaced with a 4tb (hence the parity swap out that started this insanity). I'll do that after the rebuild finishes.. a NON correcting check (preserve the crime scene so to speak). I still think that the weirdness happened when I did the parity swapout, that the parityu build did a spindown in there somewhere. I suppose I could 'replace' the parity drive again (still have 3 pre-cleared 4's here) with spindown turned OFF and see what happens there with a build/check.. but lord that's another 36+ hours

 

 

One other interesting tidbit r.e. this issue => it seems it only happens with v6  [At least I don't recall anyone reporting it with v5; and those who have experienced it generally have some XFS drives so they can't try it with v5].

 

This may simply "go away" with a future update (i.e. different kernel, driver mods, etc.)

 

V6 is the first time we have had mixed file systems. It could be that mixed file systems cause this?

V6 is the first time we have had mixed file systems. It could be that mixed file systems cause this?

It should not as at the level of parity check unRAID has no understanding of file systems.

Yes I know. But has anyone has this issue with all rfs drives?

 

Just looking for patterns and linkages in the affected systems.

 

Until resolved all identified patterns and linkages are possibly suspect, no matter how remote.

Yes I know. But has anyone has this issue with all rfs drives?

 

Just looking for patterns and linkages in the affected systems.

 

Until resolved all identified patterns and linkages are possibly suspect, no matter how remote.

 

Once you eliminate the impossible, whatever remains, no matter how improbable, must be the truth.

 

-- Arthur Conan Doyle

 

It does seem fairly unlikely the file system matters; but anything's possible.  I'd think it more likely it's an interaction with the new GUI or something strange in the Linux kernel.  But even if one of those is contributing to the issue, it's not the sole cause -- otherwise everyone would be seeing this problem (which would, of course, make fixing it MUCH easier).

 

allrighty

 

the good news is that the parity check after enlarging the new drive came up with 0 errors

... the bad news is I forgot to re-enable spindown for the parity drive (DOH!)

 

well I have a another drive to expand, doing that in the morning (getting grief from the other half on "when can I actually WATCH something again?" been keeping things from accessing it when trying this stuff... so... report later....