no1warr1or

The first error to pop up is d3hot to d0 which is why I think/thought it was ASPM related. After that error pops up the card spits out the rest of those errors and more. I have IT firmware .12 that was released in the truenas forums that was supposed to help with the card going through resets. Which it did. But I still get these weird errors shortly after restarting the system. It can go many months without an error, but the minute I restart the systems it's almost a guarantee it'll pop up. Seems like these 3 series cards are filled with problems, I see a lot of posts related to them. Looking at system drivers there are already tweaks that unraid made to it to fix an issue with other 3 series cards. Ultimately I moved my array back to an old case and board with a 2 something series lsi sas card and rebuilt my array. Waiting for a unifi unas to come in along with a handful of 26TB drives to copy my data. Ill decide the fate of this server later.

75% into the parity rebuild and it crashed with the same error. I still cant believe the OS decided to disable 2 of my disks and theres no option to enable without rebuild. So here we go again starting with ANOTHER rebuild. I reseated the card again, disabled ALL cstates, hooked up aux power to the LSI card and now I wait to see if it can manage a rebuild before a crash or before unraid decides to disable another drive for no reason and I lose 60TB of data

Not yet, currently in the middle of parity sync to restore parity 1 and disk 1. Once that's completed im gonna dig further.

I've reseated the card a couple times over the last year or so with this issue. It has a 90mm noctua fan on top of the heatsink on full speed. The card can get hit hard transfer speed wise when doing rebuilds without issues and stay running for months is what gets me, but as soon as I do an update/reboot its a gamble if this pops back up.. I've considered maybe the aux 6pin power might need to be hooked up but I dont have a fully populated card ( only 3 sas connectors in use ). Ive also considered manual link speed for PCIe, but I hadnt thought of changing slots. Right now I'm worried about changing anything, With the OS deciding to disable Parity 1 and Disk 1 I need to get the array back in order. Currently running Extended SMART tests to be safe per the guide for restoring disabled drives.. But I know its just because of this issue and I honestly cant believe I have to rebuild the drives to re-enable them.

Ive posted a couple times related to issues with my LSI card entering a low power state and not coming back (d3hot to cold or something along those lines, once i get back to my laptop ill pull the logs) But its gotten to a point where I need to spend the time to ensure it gets solved, because its plaguing my main storage server. The issue mainly seems to pop up after updating unraid (version doesnt seem to matter), after an update it can pop up after a few minutes to a day later, its random, and after so many times in a row doing it eventually it just doesnt do it anymore and stays on. This time after updating from rc2 to release it had the same behavior but this time there were so many array errors it disabled 2 of my disks. I have disabled all PCIe power management in the bios, disabled everything power management except for basic c states even disabling c6 due to reports of errors. Updated the bios, updated to the latest IT firmware on the LSI card they dropped in the TrueNAS forums. I have added in comments to boot "pci=noaer pcie_aspm=off" I dont understand why its still dropping into a low power state. System Specs: asrock Z690 Pro RS w/ Intel i5-12400 LSI 9300-16i The first couple errors before the system starts spitting out disk errors: Bottom to Top - Blockbuster kern kernel mpt3sas_cm1: basefault_reset_work: Running mpt3sas_dead_ioc thread success !!!! Blockbuster kern kernel mpt3sas_cm1: SAS host is non-operational !!!! Blockbuster kern kernel mpt3sas_cm0: SAS host is non-operational !!!! Blockbuster kern kernel mpt3sas_cm1: SAS host is non-operational !!!! Blockbuster kern kernel mpt3sas_cm0: SAS host is non-operational !!!! Blockbuster kern kernel mpt3sas_cm1: SAS host is non-operational !!!! Blockbuster kern kernel mpt3sas_cm0: SAS host is non-operational !!!! Blockbuster kern kernel mpt3sas_cm1: SAS host is non-operational !!!! Blockbuster kern kernel mpt3sas_cm0: SAS host is non-operational !!!! Blockbuster kern kernel mpt3sas_cm1: SAS host is non-operational !!!! Blockbuster kern kernel mpt3sas_cm0: SAS host is non-operational !!!! Blockbuster kern kernel mpt3sas_cm1: SAS host is non-operational !!!! Blockbuster kern kernel pcieport 0000:02:08.0: Unable to change power state from D3hot to D0, device inaccessible

Same issue here...

And it just locked up again.. sigh

I think this may be the issue im having as well with both an integrated NIC and a PCIe NIC. Both went offline and stopped showing anything 2 days ago in unifi until I hard rebooted the system tonight. Before this I was having issues with the PCIe switch in the mobo going to sleep and not waking up breaking the PCIe link to my LSI SAS card. Aspm and all that is disabled in the bios. Unfortunately I have no syslogs from the previous boot.

For some reason 2 days ago the server went completely offline and I just noticed today.. unfortunately im not able to access the last log file as it seems the stable doesnt include that functionality and only provides the log file of the current boot. I got no notifications via discord which I have setup so not a clue what happened. Machine was still physically on so I had to hold the power button to shut it down...

Update, so far it's been stable since I updated the bios and changed those settings. For anyone looking this is an asrock z690 Pro RS board, was on 18.00 on 20.01 now. Then the above mentioned bios changes.

Thanks. Interestingly ASPM shows disabled when I run the LSPCI -vv command. But power management shows enabled with all the different D states

Update- I updated the bios. All the settings I mentioned were disabled already, I enabled pcie native control. I also disabled above 4g decoding and C.A.M which is asrocks version of resizable bar. I also switched the PCIe link speed from auto to gen3 (it's a Gen 5 slot) not sure if any of this matters but will monitor.

I run unraid 7 currently. During 7.0.0 RC2 I received notifications regarding array errors. Checking the logs before these errors I saw "Unable to change power state from D3cold to D0" and similar with D3hot. The device this error referred to is the PCIe switch itself on the mobo (PEX 8724) that my LSI card is plugged into. After it happened on RC2 I rebooted and it never popped back up. Now that I'm on 7 release it keeps occurring, to the point where my drives aren't able to unmount when restarting. Anyways the plan is to update my bios, and then disable the following in the bios. -PCIe native control -PCIe aspm support -PCH PCIe aspm support -DMI aspm support -PCH DMI aspm support I'm curious if there's a way inside unraid to prevent this from happening before I'm able to sit with the machine. Also any other advice would be appreciated.