Parrity sync errors, always 5, not marvell

[adolfotregosa]

Hello. 

 

I'm scratching my head over this one. Not marvell controller, build in asmedia 1166. It is a qnap ts-664 with 16Gb ram. I never ever got an error shown in the main tab, always zero errors but I always get 5 sync errors when doing the parity, always the same sectors, identical sectors to what has happen to other people around here in the past but they seam to be using marvell controllers at that time.

 

May  2 11:15:54 unRaid kernel: md: recovery thread: P corrected, sector=3519069768
May  2 11:15:54 unRaid kernel: md: recovery thread: P corrected, sector=3519069776
May  2 11:15:54 unRaid kernel: md: recovery thread: P corrected, sector=3519069784
May  2 11:15:54 unRaid kernel: md: recovery thread: P corrected, sector=3519069792
May  2 11:15:54 unRaid kernel: md: recovery thread: P corrected, sector=3519069800

 

Basically I have no idea on how to fix this one. There is nothing to mess within the bios and messing around with tunable(s) is not fixing the issue.

 

Any suggestions ??

 

Thank you so much.

 

 

unraid-diagnostics-20230502-1129.zip

[JorgeB]

Does it only happen after a reboot or it does it find the same errors in two consecutive checks without rebooting?

[adolfotregosa]

I'm currently double checking that. In the beginning I did saw parity check run after rebooting until I changed Shutdown time-out and

VM shutdown time-out to a larger value, but I'm almost sure I already run 2 parity checks without rebooting and got the same 5 sync errors. Anyhow, if this parity check does not produce errors but they re-appear after reboot when the reboot seams to be clean, what can I do ?

 

Does the parity check need to go all the way to the end to actually correct errors ? 

[JorgeB]

Either way this could be a controller or device issue, I would re-test after making a new array without both SSDs, if still issues repeat with just the SSDs and parity, this would rule out all the devices except parity.

[adolfotregosa]

Wait, I have 3 ssd and 1 nvme. What should I do? Keep only the spinning rust?

[JorgeB]

Yes, missed the other two, do a new array with only the disks, re-test, if the same repeat with only the SSDs, and the parity disk or use on SSD as parity if it's empty.

[itimpi]

38 minutes ago, adolfotregosa said:

Does the parity check need to go all the way to the end to actually correct errors ?

You need to run a correcting check (when you run such a check it will report the same number of errors as it corrects them) - were you doing this earlier?    Your diagnostics only showed the one (correcting) check so it is not possible to check from them whether earlier ones were correcting or not.  . Any subsequent checks after a correcting one should then report 0 errors.

[adolfotregosa]

Just now, itimpi said:

You need to run a correcting check (when you run such a check it will report the same number of errors as it corrects them) - were you doing this earlier?    Your diagnostics only showed the one (correcting) check so it is not possible to check from them whether earlier ones were correcting or not.  . Any subsequent checks after a correcting one should then report 0 errors.

 

Yeah I know. I'm almost sure I did that yesterday, twice, but then I rebooted. It is currently doing another pass to see if they pop up. If they do not, I'll do another pass. If still nothing I'll reboot and then my gut say they are coming back. I'll report when it finishes all the steps.

[adolfotregosa]

Ok. 

I confirm that the sync errors were corrected. I stopped the array, started and re run parity check. Still zero errors. 
 

I then rebooted and the 5 errors came back. 

diagnostics 0750 has 2 parity checks, zero errors. I stopped the array between checks. 
 

Diagnostics 1252 is after reboot. 5 errors.

 

What do I do now? 

unraid-diagnostics-20230503-0750.zip unraid-diagnostics-20230503-1252.zip

[trurl]

SSDs in the array cannot be trimmed, and can only be written at parity speed.

 

I wonder if those Crucial MX500s might be involved in this.

[adolfotregosa]

5 minutes ago, trurl said:

SSDs in the array cannot be trimmed, and can only be written at parity speed.

 

I wonder if those Crucial MX500s might be involved in this.

I have no idea why would they but… I’ll wait for instructions from the masters. I confirm the issue only comes back after a reboot. 
 

Write speed is not an issue because of cache. 
 

What I want to see is the moment the sync errors appear. In my case I’m wondering if it is at 50%, when the smaller 2Tb ssd stops doing parity.

Edited May 3, 2023 by adolfotregosa

[JorgeB]

On 5/2/2023 at 12:38 PM, JorgeB said:

do a new array with only the disks, re-test, if the same repeat with only the SSDs, and the parity disk or use on SSD as parity if it's empty.

 

[adolfotregosa]

ok, so far as testing goes, it seams that EVERY time AFTER reboot, no matter what disks are selected, I will get sync errors. The only sniping rust results came with 2 sync errors and they coincidentally are always the "same sectors", is this case:

 

May  2 11:15:54 unRaid kernel: md: recovery thread: P corrected, sector=3519069768
May  2 11:15:54 unRaid kernel: md: recovery thread: P corrected, sector=3519069800

 

The parity drive seams to be fine, I think. I mean, if I don't reboot and keep checking parity it always comes with zero sync errors after they are "corrected".

 

basically after every reboot I get sync errors. The "error sectors" are strangely coincident with what other people have gotten in the past. I dunno. Bug ?

 

I am now in the process of doing ONLY the 2TB ssd with the 4TB parity hdd and just for my knowledge, Is it normal that the parity sync keeps going after the 2TB  ssd is no more? Probably! But what is it syncing ? air ? 

 

So what now ?? lol help !

Edited May 5, 2023 by adolfotregosa

[JorgeB]

9 hours ago, adolfotregosa said:

Is it normal that the parity sync keeps going after the 2TB  ssd is no more?

Yes.

[adolfotregosa]

Well I give up. No matter what disks I choose I always get sync errors after a reboot. Same sectors coincidentally to what other people have had.  

 

May  2 11:15:54 unRaid kernel: md: recovery thread: P corrected, sector=3519069768
May  2 11:15:54 unRaid kernel: md: recovery thread: P corrected, sector=3519069800

 

I even took the risk and updated the asmedia 1166 firmware, no dice. If this is not an unraid issue and hardware is fine, I dunno what more to do. Is there anything else to do ?? Parity is fine until I reboot. I can stop and restart array, no issues, but if I restart sync errors appear. 

[JorgeB]

22 minutes ago, adolfotregosa said:

No matter what disks I choose I always get sync errors after a reboot.

That would point to a controller issue.

[adolfotregosa]

10 minutes ago, JorgeB said:

That would point to a controller issue.

“”Probably”” but I’m not going to assume that a brand new qnap is broken and the asmedia 1166 does not seam to have issues with unraid looking throughout the web so I’ll have to find alternatives to unRaid, kinda gutted on this one. I’ve been using unraid for such a long time now, oh well. 

Edited May 5, 2023 by adolfotregosa

[adolfotregosa]

managed to borrow 3 identical 3Tb hdd and retested with only those 3 hdd.

 

After reboot:

May 14 03:58:43 unRaid kernel: md: recovery thread: P corrected, sector=1565565768
May 14 03:58:43 unRaid kernel: md: recovery thread: P corrected, sector=1565565800

 

Same sectors like before ??

 

In the mean while I've been using mergerfs with snapraid and while I understand it's a different working principle, I can reboot as many time as I want and parity check always comes clean.

I cannot understand how can this be an hardware error, I'm sorry. If i don't reboot unraid parity check always comes clean.

 

I gave it my best shot.

[adolfotregosa]

Just a follow up. 

 

I realized I did not try ONE last combination after all this time. Use one of the NVME slot for the parity!  and wouldn't you know... it actually stops the 5 parity errors. It's the exact same hardware, same hdds, sdds and nvmes, same everything, just swapped the disk functions. The nvmes are not connected to the asmedia chip but the other 6 HDD are, just like before. I do not understand why this works, but it works . Any suggestions ?

Edited July 17, 2023 by adolfotregosa

[JorgeB]

3 hours ago, adolfotregosa said:

I do not understand why this works

Same, kind of strange.

[adolfotregosa]

20 minutes ago, JorgeB said:

Same, kind of strange.

You still think it is not some kind of bug? Really strange this one. I even got it while using unraid on top of proxmox with hdds in pass-through, not the controller itself!! I could stop and start unraid VM no issues but if I rebooted proxmox the issue appeared. I did the nvme experiment while I was in proxmox and just now with unraid in bare metal. Both situations gave no parity errors as long the asmedia was not used for parity.

[JorgeB]

1 minute ago, adolfotregosa said:

You still think it is not some kind of bug?

Not really, cannot see how this can be an Unraid bug, or there would be a lot more cases, IMHO it must be hardware related.

[adolfotregosa]

Just now, JorgeB said:

Not really, cannot see how this can be an Unraid bug, or there would be a lot more cases, IMHO it must be hardware related.

Hardware or not it is working correctly this way so kinda strange to think it's still hardware. Same hdds, same controllers just different functions.  Oh well I'll keep and eye on it.