Hi there,

I run monthly (non-correcting) parity checks. My monthly ran last week, and the push notification to my phone reported it as finding 0 errors.

However, I logged into the GUI today to see the dashboard reporting that there were actually 1331 errors found.

This is the first time that I've received any parity errors.

So this post is two-fold: 

a) to highlight a discrepancy in the notofication for a completed parity check; and 

b) to ask what I should do next.

Should I run a correcting parity check? Before I do, though, Is there any way to find out what sectors the parity errors were part of (to narrow down which disk/data may be at risk)?

What are some of the possible causes for these errors? ie how can I figure out why they occured on my system?

I have an old 2TB disk that I recently added to the array (disk 7), which had some 'uncorrectable' errors, but the count has remained stable at '27' since I installed it. The disk remains empty. I'm hoping they're from that, as it's an easy fix.

Attached are my diagnostics, but also screenshots showing the different parity notifications.

Thank you for you help.

willow-diagnostics-20250506-1611.zip

Trust this:

Notifications have been off for some time, but should be fixed in 7.1

Good to know, thanks.

There haven't been any unclean shutdowns since my previous parity check. What should I do next? My SMART reports all look ok.

I run nightly backups to an SSD pool, and weekly backups from that pool to an alternating pair of external drives.

Is there any way I can see what data lies at the affected sector, and then check that data (if it's important) against the backups? All disks are XFS, and I don't run the data integrity plugin.

Many thanks.

2 minutes ago, jademonkee said:

any way I can see what data lies at the affected sector

no

Beware that to an yet unfixed bug, the notification mail/msg shows the results of the semi last run.

They say, they have fixed it in 7.1

We will see...

 

1 hour ago, trurl said:

no

So is it best to just fix parity and hope for the best?

I can't see any obvious reasons to see why it happened, which irks me.

Thinking on a possible cause:

I removed the HBA card when I was booting to bare metal Windows to test my optical drive (documented here), so I wonder if it's perhaps not seated correctly, either in the PCIe slot, or if a cable has come loose. I did that on April 22nd, so between the last "0 error" parity check and this most recent one, so it's the most likely occurance I can think of.

But wouldn't that throw CRC errors when it goes to write/read to the disks?

I'm running another (non-correcting) parity check, but when it completes I'll check that the HBA card + cables are properly seated.

I might see if I can generate a list of all files written to the array between the last "0 error" parity check and today, and restore them from backup, just to be sure.

Edited May 7, 20251 yr by jademonkee

Update:

I have created a list of all files on the array that were modified since the last succesful parity check using the command:

 

find /mnt/disk1/ -newermt "4 April 2025" -not -newermt "7 May 2025" -type f >> modified_files.txt

(I ran it for each disk in the array)

 

So I now have a list of potentially problematic files that I will keep for future reference.

Of the affected files, there are a few that I care about and have backups for. In addition to my backup disks, these are also stored on an SSD passed through to my VM. This SSD is the "working directory" for those files, which I then copy to the array. So I have a copy of each of these files that have never touched the array, and are therefore uncorrupted.

As the SSD is passed through to my Windows VM, I used the Windows program FreeFileSync (which I also use to copy the files to the Unraid array) to run a CRC compare of the versions of the affected files stored on the SSD vs the array. There were no differences found (yay!).

The Uncorrecting Parity check is still running, but I'll let it finish (it's still yet to find the final 4 errors - count is currently at 1329), then I'll re-seat the LBA card and cables, and run a correcting parity check. I'm comfortable with any of the remaining files having potential corruption.

Though I might run filesystem integrity checks beforehand just to be sure.

Thanks all for your input.

(likely) Final update:

The parity check finished with 1329 errors this time (I'm assuming something was overwritten between the previous check and today's).

I shutdown the server and checked the LBA card + cables. Nothing was obviously loose, but I can't say definitively.

After booting, I mounted the disks in maintenance mode and ran filesystem checks on them. No errors found. I'm also running scrubs on my two ZFS pools, just to check.

After the scrubs have completed, I will run a manual parity check with "write corrections" enabled. I have a feeling that the cause was probably the removal and reinstallation of the LBA card, as I can't think of what else could be responsible. Hopefully the errors were all in Parity and not the Array disks.

Edited May 7, 20251 yr by jademonkee

1 hour ago, jademonkee said:

run a manual parity check with "write corrections" enabled

And then it might be a good idea to run another noncorrecting check to verify that no parity errors remain. If it still has parity errors after correcting them then you still have a problem.

 

Make sure you get diagnostics or at least syslog for each of the parity check so they can be compared.

18 hours ago, trurl said:

And then it might be a good idea to run another noncorrecting check to verify that no parity errors remain. If it still has parity errors after correcting them then you still have a problem.

 

Make sure you get diagnostics or at least syslog for each of the parity check so they can be compared.

 

Correcting parity check completed with 1324 errors (so it keeps decreasing).

I'm now running a non-correcting parity check to see if any more appear.

I have collected a diagnostics for posterity.

Thanks again to everyone for your help.