So for a little bit of context I recently flashed a new bios on my motherboard to get it ready for an upgrade I was planning on doing, new cpu and ram. I thought everything was working but after about 6 hours my cpu hard locked up and became unresponsive while transferring some files. I was unable to get any logs of what happened, but after looking at some other forum posts I thought it might be not having reset some of the settings on the motherboard after the bios flash.  I have now hopefully set the bios up properly with “power supply idle control” set to “typical current idle” and disabling the C-states globally.

 
I went to run a parity check after it was stable for about a day only to have it completely crash at around 40ish%. I also unfortunately didn’t get the logs for that either. Upon reboot I saw that both my parity drives have been disabled and figured I would just rebuild them. However….

Now when I try to rebuild the drives I am getting “Alert [CUBE] - parity2 in error state (disk dsbl)” and “Alert [CUBE] - parity in error state (disk dsbl)” along with a warning that 6 of my disks have read errors.

 

Any ideas of how to get this fixed and my server back to running normal again? Hopefully saving as much of the data as possible.

I will try to be as responsive as I can while at work.

I uploaded 2 diagnostics files. one is before the second failed attempt, the other after with the errors. 

 

 

For reference my system is a

AMD Ryzen 5 3600 6-Core @ 3600 MHz
ASRock B550M Steel Legend
American Megatrends International, LLC., Version P2.80
BIOS dated: Fri 05 May 2023 12:00:00 AM MDT
with 32gig of ram

I am running version 6.11.1

cube-diagnostics-20230823-2351.zip cube-diagnostics-20230823-2318.zip

Aug 23 23:27:45 Cube kernel: mpt2sas_cm0 fault info from func: mpt3sas_scsih_issue_tm
Aug 23 23:27:45 Cube kernel: mpt2sas_cm0: fault_state(0x2622)!
Aug 23 23:27:45 Cube kernel: mpt2sas_cm0 fault info from func: mpt3sas_base_make_ioc_ready

 

HBA problems, make sure it's well seated and sufficiently cooled, you can also try a different PCIe slot if available.

6 hours ago, JorgeB said:

Aug 23 23:27:45 Cube kernel: mpt2sas_cm0 fault info from func: mpt3sas_scsih_issue_tm
Aug 23 23:27:45 Cube kernel: mpt2sas_cm0: fault_state(0x2622)!
Aug 23 23:27:45 Cube kernel: mpt2sas_cm0 fault info from func: mpt3sas_base_make_ioc_ready

 

HBA problems, make sure it's well seated and sufficiently cooled, you can also try a different PCIe slot if available.

I popped the card out and it seemed the thermal paste on it turned to a dry cement.  Seems like it could be overheating. I gave the card a good clean and reapplied new paste. I reseated it a few times to make sure the pins were making better contact as well.  I’ll give it a shot and see if that was the culprit.

It looks like its running for a little bit but then will start to hit many of my cpu cores to 100%. I noticed in my log that it seems the mpt2sas_cm0: fault_state(0x2622)! keeps happening resets and then will fault again.  
Did i miss something in the bios?
Could it be that the card is failing and i should just replace it?

cube-syslog-20230824-1659.zip

Edited August 24, 20232 yr by gjkthx
typo

1 minute ago, gjkthx said:

Could it be that the card is failing and i should just replace it?

Could be, can you try it in a different PCIe slot or PC?

58 minutes ago, JorgeB said:

Could be, can you try it in a different PCIe slot or PC?

I pulled the gpu and moved it into that slot since it was the only other one that could hold it.  Once it was on it seemed to have set 2 of the other disks to disabled. The cable that those are on has 4 drives and the other 2 show up.  I replaced the cable with a spare I had to rule it out, but now I can’t enable those disks. When I try to run a check on them I get

Phase 1 - find and verify superblock...

superblock read failed, offset 0, size 524288, ag 0, rval -1

fatal error -- Input/output error

While the SMART reports on them still seem to be fine.

I feel as though I have made it worse.

I did move it back down to another slot just so i could get into the bios if needed.

cube-smart-20230824-1214.zip cube-smart-20230824-1211.zip

I made a new config and started the parity reconstruction again this time with the card in the top pci slot. It so far seems to be alright and hasn’t crashed yet. So maybe that pci slot is gone for me. On a side note, I have noticed thought that I will hit 100% usage on one of my cores for 20 seconds its always followed by

Aug 24 14:00:51 Cube kernel: usb 1-6: reset high-speed USB device number 3 using xhci_hcd

Aug 24 14:00:52 Cube kernel: sd 0:0:0:0: [sda] 62656641 512-byte logical blocks: (32.1 GB/29.9 GiB)

Aug 24 14:02:32 Cube kernel: usb 1-6: reset high-speed USB device number 3 using xhci_hcd

Aug 24 14:02:33 Cube kernel: sd 0:0:0:0: [sda] 62656641 512-byte logical blocks: (32.1 GB/29.9 GiB)

Aug 24 14:06:06 Cube kernel: usb 1-6: reset high-speed USB device number 3 using xhci_hcd

Aug 24 14:06:06 Cube kernel: sd 0:0:0:0: [sda] 62656641 512-byte logical blocks: (32.1 GB/29.9 GiB)

Any idea of what would be causing this?

Try plunging the flash drive in a different port, ideally a USB 2.0 port.

11 hours ago, JorgeB said:

Try plunging the flash drive in a different port, ideally a USB 2.0 port.

I might have corrupted a file while it was turning off and on during the trouble shooting.  I should have that fixed now. It also turns out that both sas card and pci slot are still good, but my case seems to pull the card out at an angle when I screwed it down. I have no idea what happened to cause this, but I now have it fixed. You have been a massive help and I can’t thank you enough. THANK YOU.