Hi there,

 

my monthly Parity Check just got through and found 132 errors. The check it setup to not correct errors, as I read this could also do more harm than good; it was then advised to use the Dynamix File Integrity plugin instead, to also see which files have issues, so I installed that one; but I can neither figure out how it works, nor a manual/tutorial  the unraid docs for the parity are also a bit lackluster what to do in case of error.

What am I supposed to do now?

 

Recently the server crashed and I rebooted, it did the Parity check and found nothing; so the 132 errors should be fairly new, the crash was around 2 weeks ago.

 

Best Regards

lunas-diagnostics-20250107-1335.zip

File integrity plugin would only help if there were already checksums before the errors, any recent unclean shutdowns?

Hi there JorgeB,

 

only that crash, which was on the from 21.12. to 22.12. last year. 

Ok, so I should setup the Integrity Plugin in general, but it doesn't help now (still, how do I set it up?), but how would I find out which files are incorrect now, so I can at least manually try to check them?

 

3 minutes ago, CameraRick said:

find out which files are incorrect now

No way to know

Hi there trurl,

 

I had a look at that thread before, I'm probably too dense but I really don't understand how to use it

 

Quote

No way to know

Hm ok... so, I do the Parity check which takes some serious time, to know that there is an error, but now I can't fix the errors because I don't know where the error sits? I'm a bit confused, what am I supposed to do now?

Unclean shutdown sometimes results in a (relatively) few parity errors such as you have.

 

Run a correcting parity check.

I did a parity check right after that one crash, since then I only ever shutdown the server (I didn't change the hardware, only installed some new fans). In that check directly after the crash, I didn't had any errors. So would it still be wise to run a correction now?

 

And what would I do in the future? It somehow feels like the parity check is not really useful if it can only tell me that there are errors, but not what is wrong; I'm not sure why that isn't a feature tbh, I always thought that would be the sense behind the check

If the errors are due to an unclean shutdown, parity is the one that will be wrong, just run a correcting check, you then run a new one to confirm all is OK.

But I can't know if it is because of an unclean shutdown, that's the issue  because after the last unclean shutdown, there was a Parity Check, and that one went with 0 Errors; the new ones came since then. That's my issue, because I can't know

 

And because I don't what Data is potentially corrupt and just learned the Parity Check doesn't really help, I kinda feel unsafe for my data

Parity check could never help correct/confirm any data corruption, you can have checksums, or use btrsf or zfs to detect/confirm if the data is OK, not correct, but that will only help for the future, for now, if you are not sure if it was the unclean shutdown, run another parity check, if it finds the same exact errors, it's most likely a parity problem, extremely rare for a disk to just corrupt data, the so called bit-rot, so if that is the case, just correct parity, if it finds a different number of errors, the RAM would be the main suspect, in that case run memtest.

2 hours ago, JorgeB said:

run another parity check, if it finds the same exact errors, it's most likely a parity problem, [...] just correct parity, if it finds a different number of errors, the RAM would be the main suspect

I'm confused - so shall I now run a parity check without the correction, see if the number is identical, and then run it again with a correction? That sounds like an incredible stress on the disks, no?

 

Quote

in that case run memtest.

I use ECC memory and Memtest never shows me errors, even though I learned that ECC (not fully buffered) is also prone to fail. I already bought new RAM twice for this machine, it's never going to be happy with the RAM choice, is it :')

10 hours ago, CameraRick said:

I'm confused - so shall I now run a parity check without the correction, see if the number is identical, and then run it again with a correction? That sounds like an incredible stress on the disks, no?

If you want to confirm the errors are being caused by another thing other than just an out of sync parity drive, it's the best way.

 

10 hours ago, CameraRick said:

I use ECC memory and Memtest never shows me errors

In that case unlikely that it's a RAM problem, most likely parity is just out of sync, or there's a bad disk, which, while extremely rare it has happened before, running the multi+ checks should confirm if that is the problem or not.

 

 

What are "multi+ checks"?

 

This night, I started a Parity-Correcting-Check; as I can't tell what files are wrong anyway, I'm not sure what else I can do to save data (as it seems I can't). Also, I ordered new Hardware to save some power, and I need to have that rebuild till next week; so yeah.

 

Once the Parity Check is done, I guess it would be wise to use the Integrity Plugin to know which data is actually cursed next time; I still have no idea how to use it, but I will probably just ask in that forum thread

8 minutes ago, CameraRick said:

Once the Parity Check is done, I guess it would be wise to use the Integrity Plugin to know which data is actually cursed next time; I still have no idea how to use it, but I will probably just ask in that forum thread

An alternative would be to switch drives to using btrfs or zfs file system types as these have built-in check-summing.  However be aware that at the moment the performance of zfs on drives when used in the main array is not very good (OK in pools).

2 hours ago, CameraRick said:

What are "multi+ checks"?

Typo, meant multiple checks.

 

 

6 hours ago, itimpi said:

An alternative would be to switch drives to using btrfs or zfs file system types as these have built-in check-summing.  However be aware that at the moment the performance of zfs on drives when used in the main array is not very good (OK in pools).

Doesn't ZFS need to have all disks spin up when there's array access? Either way, I don't have enough external storage to get my Data off the array, re-format and copy everything back, so I simply can't do that. I am fine with using the Integrity Plugin, now that I know it's kinda mandatory for knowing what files corrupted; and I have to re-evaluate if I do Parity checks in general, they really don't do as I thought they would, like this it's not very useful to me. Might sound really stupid but I am really disappointed by this

 

4 hours ago, JorgeB said:

Typo, meant multiple checks.

Ah I see! Time's running a bit, so I can't really do so many checks these days, as they need around 2 days each (and it's not always in sync with my sleeping schedule, ha). But as this doesn't give any more intel, I am fine with the corrective check now and will try that plugin after 😮

On 1/7/2025 at 3:08 PM, JorgeB said:

File integrity plugin would only help if there were already checksums before the errors,

Thinking about this again - 

 

Would it be helpful to have Checksums of all files anyway? Because then it will probably "find" the faulty files?
As the Checksum from the Array and the Parity are not the same, it can't tell me which are right, but it can at least tell me which files queue the error, no?

4 minutes ago, CameraRick said:

and the Parity are not the same,

I think you misunderstand how parity works.   It contains no data and therefore no checksums.

36 minutes ago, CameraRick said:

Parity checks in general, they really don't do as I thought they would, like this it's not very useful to me

They warn you when you may have issues.

8 minutes ago, itimpi said:

I think you misunderstand how parity works.   It contains no data and therefore no checksums.

I am aware they don't contain data, but I thought you could probably "extrapolate" the Checksums from it. My bad!

So the Dynamix Integrity Plugin works by storing the Checksums in a "text file", and comparing them against the files themselves I guess? Makes sense

 

2 minutes ago, itimpi said:

They warn you when you may have issues.

Yeah, but it's a big "IF" with no possible solution to know afterwards, that's my concern. I don't know which files, I don't know if it's the Parity or the Files... and if it is because of a failing Drive, the many additional checks only stress the failing Drive(s) more. I'm just not well-versed enough with these things to wrap my head around what to make of it, and what to do with the information.

 

btw, the currently running, correcting check is at ~70% and so far found 135 Issues, so three more than before. Last time I got the Issues very early, but the rest went through Ok, so I don't think more issues will arise

5 hours ago, CameraRick said:

so three more than before

Make sure you have syslogs (or diagnostics) for both parity checks so they can be compared.

I just downloaded the new one, the one in the first post in this thread was after the former one (obviously, haha)

But what can we compare here, if we can't tell which files are "faulty"?

lunas-diagnostics-20250109-0936.zip

Log stops logging the errors after a few, but you can already see a new one:

 

Jan  6 00:00:21 LuNAS kernel: md: recovery thread: PQ incorrect, sector=0
Jan  6 00:00:21 LuNAS kernel: md: recovery thread: PQ incorrect, sector=32
Jan  6 00:00:21 LuNAS kernel: md: recovery thread: PQ incorrect, sector=48
Jan  6 01:55:48 LuNAS kernel: md: recovery thread: PQ incorrect, sector=1489890168

Jan  7 23:11:46 LuNAS kernel: md: recovery thread: PQ corrected, sector=0
Jan  7 23:11:46 LuNAS kernel: md: recovery thread: PQ corrected, sector=32
Jan  7 23:11:46 LuNAS kernel: md: recovery thread: PQ corrected, sector=48
Jan  7 23:55:20 LuNAS kernel: md: recovery thread: PQ corrected, sector=754388272
Jan  8 00:42:12 LuNAS kernel: md: recovery thread: PQ corrected, sector=1489890168

 

This suggests a hardware issue, RAM is the usual suspect, you say that you are using ECC, but it's a Ryzen CPU, ECC support can be spotty from what I've read, so RAM would still be my main suspect, and if it's not RAM it's a disk, and those are much rarer but happened before, and it's much more difficult to troubleshoot, you basically need to retest without one of disks at a time.

 

 

 

Hi JorgeB,

 

I see! I actually do think that the RAM is spotty as well. I'm currently in the middle of reworking the Server and switch to an Intel System; no ECC, but yeah. I had three modules in my AMD system, I am currently re-using just one of them (a single 32GB module, which is the newest one). I will probably switch to a new pair entirely.

 

Some of the guys in the german forums now hammered some sense into me, so I also ordered some new HDDs for external connection to make some proper backups of the data. So, the future should hopefully get these issues out

 

Best Regards