I have a amd threadripper machine that keeps on freezing whenever the parity check runs. I am stuck on what else I can do.

I followed all the threads on trying to disable any C-states within the bios(I found the C-states disabling makes no difference). I replaced the power supply, checked all the memory, and even replaced the sata controller with a new 16 port lsi 9300. I updated the motherboard bios to the latest F6 version.

And I updated the LSI logic controller to the latest versions available.

The machine is running with 14 disks( 6 disks are part of the main array a 62 TB) and a separate ssd cache pool, and about 15 active docker containers (like a mysql db) that have lots of disk access, but the machine is very stable for over a week if I turn off the parity check. As soon as the parity check runs it just freezing randomly (some times shortly after starting or could be half way or more)

 

Specs:
Gigabyte Technology Co., Ltd. TRX40 AORUS PRO WIFI , Version Default string
American Megatrends International, LLC., Version FD
BIOS dated: Wed 7 Sep 2022 12:00:00 AM PDT

AMD Ryzen Threadripper 3960X 24-Core @ 3800 MHz

 LSI Logic Controller SAS 9300 16-port

Memory: 128 GiB DDR4

 

 

The versions of my LSI card is :

root@media:~# lspci | grep LSI
23:00.0 Serial Attached SCSI controller: Broadcom / LSI SAS3008 PCI-Express Fusion-MPT SAS-3 (rev 02)
25:00.0 Serial Attached SCSI controller: Broadcom / LSI SAS3008 PCI-Express Fusion-MPT SAS-3 (rev 02)

 

dmesg | grep mpt3sas_cm0 

...

[    9.488835] mpt3sas_cm0: LSISAS3008: FWVersion(16.00.10.00), ChipRevision(0x02), BiosVersion(18.00.00.00)

...

 

I did notice this error in dmesg "overriding NVDATA EEDPTagMode setting" which maybe is related ?

I also tried logging to a separate syslog server to try to capture any errors just as it froze but it just suddenly stops with not errors.

 

Can someone take a look at me diagnostics and let me know if they see anything strange? 

media-diagnostics-20230926-1230.zip

Edited September 26, 20232 yr by dandiodati
Adding more info on lsi card

Crashing only during a check suggests a hardware problem, but without anything logged difficult to guess what, check cooling, including for the HBA, you can also run memtest, if you have a different PSU try it.

Thats what I heard before and I did change to a new PSU and (HBA)controller card. And I added more fans just incase for cooling and was monitoring temperatures which seem fine. 
I ran memtest and even tried taking out some of the memory sticks. I dont have a spare motherboard and replacing that it more expensive and afraid it might not fix it.

Like I said it is stable unless I run the parity check.

 

Can someone please still look at the diagnostics since I saw one item that may be an error (like noted) but not an expert on all the other messages and on which ones might be a clue on the issue.

Edited September 27, 20232 yr by dandiodati

There's nothing relevant that I can see in the diags, the error you mention, which is not really an error, is harmless.

Doh ok thank you for taking a look. Its very frustrating since things used to be very stable. Do you suggest I replace the motherboard next (getting to the point of replacing everything) ? 

Was thinking first to run with subset of the ram sticks just to see if there is any difference.  

10 hours ago, dandiodati said:

Was thinking first to run with subset of the ram sticks just to see if there is any difference.  

It's worth a try, also check cooling/temps.