Hi,

I recently changed some major components in my unraid server: mainboard, cpu, ram, HBA
I changed from an i7 4790 to a ryzen 5 3400g, 16gb ddr4, gigabyte aorus b450 and I changed my cheapo Amazon HBA for an LSI SAS2308 (exact modelnumber not in mind). Everythink worked fine, did a parrity check, put new data on it, mover worked just fine. I also got rid of any smart failures, after switching to the new quality HBA. Unraid Version 6.9.1

 

But now I get about 850 parrity errors - of course rebooted, already took the HBA out and back in again, changed the sata breakout cable - I'm clueless.....

 

I attached the diagnostics file, hopefully somebody can help me out. Already tried to read any information out of it but no chance for me

PS: Sorry for my maybe bad english

tower-diagnostics-20210330-2356.zip

Edited March 30, 20215 yr by strohann

You have to run a correcting parity check to correct those errors, then after the parity errors are corrected, you have to run a non-correcting parity check to make sure there are no parity errors. If you still have errors after you have corrected the parity errors, then post new diagnostics without rebooting so we can see both parity checks in syslog.

 

Not related, but your appdata, domains, system shares have files on the array.

Thanks for your quick response.

 

I read in a different post to not do a correcting parrity check, because any failure would then be written to the array.

Did you find the failure on the log files or did you know it because of what I wrote before? Just asking for the next time there is any failure like that. Can you explain a little bit more please?

 

@appdata etc on array: I wanted to have my files in these folders secured on my array. But I think if I want to have them secured I have to use a second 500gb cache drive and configure them in raid 1, correct?

2 hours ago, strohann said:

I read in a different post to not do a correcting parrity check, because any failure would then be written to the array.

You must have misunderstood what you read, or else whoever wrote that was mistaken.

 

Correcting parity check never writes anything to your data disks, it only writes parity so the errors are corrected. The only way to get parity errors corrected is to do a correcting parity check.

 

The usual recommendation is to have a non-correcting parity check when you don't suspect you have any parity errors, such as the scheduled parity check. Then if it turns out you do have parity errors, you can decide if there is some other reason such as a specific disk having problems and fix any problems and run another non-correcting parity check.

 

But in this case, you already know you have parity errors. The small number of parity errors you have is typical of unclean shutdown.

 

You have to correct parity errors or those errors would affect rebuilding a data disk if that became necessary. Exactly zero parity errors is the only acceptable result, and until you achieve that you still have work to do.

 

Sometimes after correcting parity errors, and then following it with a noncorrecting check to verify there are exactly zero parity errors, it will turn out that there are still parity errors. That would usually indicate some hardware problem. That is why you need to correct and then follow it with noncorrecting, and do all that without rebooting so both parity checks can be examined in syslog when you post diagnostics.

 

2 hours ago, strohann said:

@appdata etc on array: I wanted to have my files in these folders secured on my array. But I think if I want to have them secured I have to use a second 500gb cache drive and configure them in raid 1, correct?

appdata, domains, system are used by dockers / VMs. If these are on the array, the disks can't spin down because they will always have open files. And dockers / VMs will have performance impacted by slower array. The usual recommendation is to have these all on a fast pool (such as cache) and configured to stay there.

 

raid1 is one way to secure them, but backup is better, and even if you have raid1 you should backup.

 

I actually have 2 pools. Cache pool for caching user share writes, 2x500GB SSD as btrfs raid1. And "fast" pool a single 256GB nvme as XFS for those docker / VM user shares (appdata,domains,system). I schedule appdata backups using the CA Backup plugin.

 

And backup goes for your array data also. Parity is not a substitute for backups. You must always have another copy of anything important and irreplaceable.

These are answers I can work with. You are very welcome so far.

I already started the correcting parrity run last night, when it's done I'll post the new diagnostics and will re-run the non-correcting run and post again.

Here's the diagnostics after the correcting parrity check:

tower-diagnostics-20210331-2028.zip

And this is the new diagnostics report:

tower-diagnostics-20210401-1731.zip

How many parity errors on the non-correcting check?

You mean after this run? 1...

2 hours ago, strohann said:

And this is the new diagnostics report:

tower-diagnostics-20210401-1731.zip 108.5 kB · 1 download

 

Three things I notice in your diagnostics (dated 20210401-1731):

• Disk 4 has a cable problem, causing the SATA link to reset.
• The Nvidia driver is spamming your syslog so you ought to uninstall it while you're trying to fix other problems.
• The parity check is a correcting one and it corrected a lot of errors. After that you need to run a non-correcting check to see if it was successful.

1 hour ago, John_M said:

Disk 4 has a cable problem, causing the SATA link to reset.

mhh, thats bad... I'm using a new sff8078 to 4x sata breakout cable. Do you really think there's a problem in the cable? Or could it just be too long with 1 meter?
In the near future I will upgrade my case for a new one with a sata/sas backplane,

 

1 hour ago, John_M said:

The Nvidia driver is spamming your syslog so you ought to uninstall it while you're trying to fix other problems.

uninstalled

 

1 hour ago, John_M said:

The parity check is a correcting one and it corrected a lot of errors. After that you need to run a non-correcting check to see if it was successful.

The log you mentioned was a non correcting one. But instead of the expected 0 errors, it gave me 1 error.
By the way, where can I find some information about how to read/find any information in these log files?
I now started a a new correcting parrity check.

2 hours ago, strohann said:

Do you really think there's a problem in the cable?

 

The SATA link is failing and being reset, so it's a problem somewhere between the controller and the drive electronics, which includes the cable and the connectors. Connections go bad more often than silicon and are generally easier to replace. It might be a damaged contact, or it might be dirty. Try plugging it and unplugging it a few times and make sure it's properly seated at both (or, rather, all 5!) ends. I've had the same problem with breakout cables. Three work just fine but the fourth one is problematic. 1 metre isn't too long.

 

Here's what that error looks like in the syslog. Notice how it couldn't sustain a 6 Gb/s link and shifted down to 3 Gb/s.

 

Mar 28 19:19:28 Tower kernel: ata1.00: exception Emask 0x11 SAct 0x4000000 SErr 0x600100 action 0x6 frozen
Mar 28 19:19:28 Tower kernel: ata1.00: irq_stat 0x48000008, interface fatal error
Mar 28 19:19:28 Tower kernel: ata1: SError: { UnrecovData BadCRC Handshk }
Mar 28 19:19:28 Tower kernel: ata1.00: failed command: READ FPDMA QUEUED
Mar 28 19:19:28 Tower kernel: ata1.00: cmd 60/00:d0:20:29:0c/04:00:00:00:00/40 tag 26 ncq dma 524288 in
Mar 28 19:19:28 Tower kernel:         res 40/00:00:20:29:0c/00:00:00:00:00/40 Emask 0x10 (ATA bus error)
Mar 28 19:19:28 Tower kernel: ata1.00: status: { DRDY }
Mar 28 19:19:28 Tower kernel: ata1: hard resetting link
Mar 28 19:19:29 Tower kernel: ata1: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
Mar 28 19:19:29 Tower kernel: ata1.00: supports DRM functions and may not be fully accessible
### [PREVIOUS LINE REPEATED 1 TIMES] ###
Mar 28 19:19:29 Tower kernel: ata1.00: configured for UDMA/133
Mar 28 19:19:29 Tower kernel: ata1: EH complete
Mar 28 19:19:29 Tower kernel: ata1: limiting SATA link speed to 3.0 Gbps

 

How do I know that ata1.00 refers to Disk4? Well, that takes a little detective work, but there are clues:

 

Mar 28 19:17:14 Tower kernel: ata1.00: ATA-9: WDC WD80EDAZ-11TA3A0, VGHJVGXG, 81.00A81, max UDMA/133

 

and, from the very useful system/vars.txt file in your diagnostics:

 

[name] => disk4
[device] => sdb
[id] => WDC_WD80EDAZ-11TA3A0_VGHJVGXG

 

which tie the disk model and serial number, the /dev/sdX identifier, the Disk slot number and the ATA bus all together. Entries in other files will confirm that this is the case.

 

The log actually covers several parity checks in total. A non-correcting one starts like this:

 

Mar 28 19:19:23 Tower kernel: mdcmd (37): check nocorrect
Mar 28 19:19:23 Tower kernel: md: recovery thread: check P ...

 

and if it finds an error, reports it like this:

 

Mar 29 07:40:45 Tower kernel: md: recovery thread: P incorrect, sector=10671239984

 

but obviously doesn't correct it. A correcting parity check starts like this:

 

Mar 31 02:38:06 Tower kernel: mdcmd (48): check 
Mar 31 02:38:06 Tower kernel: md: recovery thread: check P ...

 

and if it finds an error, reports it like this:

 

Mar 31 14:31:48 Tower kernel: md: recovery thread: P corrected, sector=10671239984

 

and, as expected, corrects it.

 

There is indeed a final non-correcting check, finding one error, which I missed because of all the noise:

 

Mar 31 20:41:02 Tower kernel: mdcmd (54): check nocorrect
Mar 31 20:41:02 Tower kernel: md: recovery thread: check P ...

...

Apr  1 02:25:05 Tower kernel: md: recovery thread: P incorrect, sector=5682714232

 

2 hours ago, strohann said:

By the way, where can I find some information about how to read/find any information in these log files?

 

Hopefully, the examples I've included will help. One difficulty is that messages from various subsystems often overlap, which is why getting rid of the Nvidia driver noise is useful. Incidentally, that looks like this:

 

Mar 29 13:08:41 Tower kernel: resource sanity check: requesting [mem 0x000c0000-0x000fffff], which spans more than PCI Bus 0000:00 [mem 0x000c0000-0x000dffff window]
Mar 29 13:08:41 Tower kernel: caller _nv000708rm+0x1af/0x200 [nvidia] mapping multiple BARs

 

repeated over and over. One way to get familiar with syslogs is to examine the one from your own server when it's running smoothly. Boot it up and look at the syslog before starting the array. Then start the array and see what messages have been added. Then start up a docker container, and so on. I hope this helps.

 

Very detailed information, thank you! I will try to get rid of the cable errors after the Easter holidays.
The part with the 3 6 GB/s is interesting: I noticed earlier. My thought were the length of the cable or maybe electromagnetic interference (this one would suck..). Well, maybe my new case will arrive shortly so I don't have these problems anymore. If so I'm pretty screwed because the 8tb drives are shucked ones from without warranty.....
I will provide new infos and then we will see..
Happy Easter holidays

 

 

 

 

 

The specified maximum cable length for internal SATA is 1 m (it's longer for eSATA). Those breakout cables are generally available in 1 m or 50 cm lengths. I've got stocks of both and use them as appropriate - if 50 cm is long enough to reach easily I use them, so as to keep the excess cable to a minimum. One thing you could try, after reseating the bad cable, is to rearrange them. Neatly run SATA cables are bad; messy cables that don't all run together in parallel are less susceptible to crosstalk. So snip any cable ties and mess them up a little - Seriously! They seem to come in two versions - one with "figure of eight" shaped red PVC covered cable and one with pairs of transparent plastic covered cables, through which pale blue foil shielding can be seen. I've tried both and have had successes and failures with both. Remember that you can move a disk to another port on the controller card or the motherboard, either temporarily for troubleshooting or permanently. Unraid doesn't mind because it keeps track of disks by their serial numbers. Another thing that can cause disk disconnection problems is poor power delivery. In particular, power splitters and Molex to SATA power converters.

 

 

On 4/3/2021 at 7:20 PM, John_M said:

The specified maximum cable length for internal SATA is 1 m (it's longer for eSATA). Those breakout cables are generally available in 1 m or 50 cm lengths. I've got stocks of both and use them as appropriate - if 50 cm is long enough to reach easily I use them, so as to keep the excess cable to a minimum. One thing you could try, after reseating the bad cable, is to rearrange them. Neatly run SATA cables are bad; messy cables that don't all run together in parallel are less susceptible to crosstalk. So snip any cable ties and mess them up a little - Seriously! They seem to come in two versions - one with "figure of eight" shaped red PVC covered cable and one with pairs of transparent plastic covered cables, through which pale blue foil shielding can be seen. I've tried both and have had successes and failures with both. Remember that you can move a disk to another port on the controller card or the motherboard, either temporarily for troubleshooting or permanently. Unraid doesn't mind because it keeps track of disks by their serial numbers. Another thing that can cause disk disconnection problems is poor power delivery. In particular, power splitters and Molex to SATA power converters.

Thanks for these infos.

 

Now getting new errors from other drives. I first have to get rid of those...

HEUREKA! My new case arrived and I couldn't resist to start switching to it also the 2 new Seagate Exos arrived.

After switching the cases (with new backplane) I did like 3 parrity checks and they were all witzout errors. Then I switched one of the new exos against the 4TB WD blue.

I will keep an eye on the whole situation but i think the problems are gone

Thank you all very very much for your help and explanatioins, I will give 5 stars on yelp (love south park :D)