July 26, 20232 yr Hello, I have 4 2tb nvme drives (Intel 670P) in a QNAP QM2-4P-384 https://www.qnap.com/en-us/product/qm2-4p-384/pc The drives were working as a cache pool. After swapping motherboards MSI B365M PRO-VDH -> Gigabyte B760M DS3H DDR4 I can't find the drives in Unraid. They do show up in the BIOS. Thinks I've tried 1) Force the PCIE slot to 3.0 2) Update the Bios 3) Try every combination I can think of for enabling/disabling things at all related to PCIE and storage control. (Can't say I've done everything but tried to go through every option) 4) Tried to start the array because I saw that elsewhere on the forums. Maybe a bad move. Nothing on the Pool is critical though so if what was on there is lost no biggie. 5) A few other here and there's that I can think of. Let me know if anyone has any ideas. I think unraid sees the QNAP card but I don't see any errors related to anything going on (but maybe I don't know where to look) Even a troubleshooting step at this point would be helpful. I'm about to throw in a blank copy of unraid to see what's detected. After that maybe put the card into my windows computer to see what shows up. Edited July 29, 20232 yr by ausername11
July 27, 20232 yr Author Small update, Tried on an Ubuntu Live USB stick and couldn't find the drives. For sure the problem is somewhere with the QNAP card probably related to the new motherboard. lscpi output below -- 01:00.0 PCI bridge: ASMedia Technology Inc. ASM2824 PCIe Gen3 Packet Switch (rev 01) 02:00.0 PCI bridge: ASMedia Technology Inc. ASM2824 PCIe Gen3 Packet Switch (rev 01) 02:04.0 PCI bridge: ASMedia Technology Inc. ASM2824 PCIe Gen3 Packet Switch (rev 01) 02:08.0 PCI bridge: ASMedia Technology Inc. ASM2824 PCIe Gen3 Packet Switch (rev 01) 02:0c.0 PCI bridge: ASMedia Technology Inc. ASM2824 PCIe Gen3 Packet Switch (rev 01) 01:00.0 PCI bridge: ASMedia Technology Inc. ASM2824 PCIe Gen3 Packet Switch (rev 01) (prog-if 00 [Normal decode]) Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0, Cache Line Size: 64 bytes Interrupt: pin A routed to IRQ 124 IOMMU group: 11 Bus: primary=01, secondary=02, subordinate=06, sec-latency=0 I/O behind bridge: [disabled] [32-bit] Memory behind bridge: 40700000-40afffff [32-bit] Prefetchable memory behind bridge: [disabled] [64-bit] Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR- BridgeCtl: Parity- SERR+ NoISA- VGA- VGA16+ MAbort- >Reset- FastB2B- PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn- Capabilities: [40] Power Management version 3 Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+) Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME- Capabilities: [50] MSI: Enable+ Count=1/1 Maskable- 64bit+ Address: 00000000fee00278 Data: 0000 Capabilities: [80] Express (v2) Upstream Port, MSI 00 DevCap: MaxPayload 512 bytes, PhantFunc 0 ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ SlotPowerLimit 75W DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq- RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ MaxPayload 256 bytes, MaxReadReq 512 bytes DevSta: CorrErr+ NonFatalErr- FatalErr- UnsupReq+ AuxPwr- TransPend- LnkCap: Port #0, Speed 8GT/s, Width x8, ASPM L1, Exit Latency L1 <64us ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+ LnkCtl: ASPM Disabled; Disabled- CommClk+ ExtSynch- ClockPM+ AutWidDis- BWInt- AutBWInt- LnkSta: Speed 2.5GT/s (downgraded), Width x8 TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt- DevCap2: Completion Timeout: Not Supported, TimeoutDis- NROPrPrP- LTR+ 10BitTagComp- 10BitTagReq- OBFF Not Supported, ExtFmt- EETLPPrefix- EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit- FRS- AtomicOpsCap: Routing- DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR+ 10BitTagReq- OBFF Disabled, AtomicOpsCtl: EgressBlck- LnkCap2: Supported Link Speeds: 2.5-8GT/s, Crosslink- Retimer- 2Retimers- DRS- LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis+ Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance Preset/De-emphasis: -6dB de-emphasis, 0dB preshoot LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete- EqualizationPhase1- EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest- Retimer- 2Retimers- CrosslinkRes: unsupported Capabilities: [c0] Subsystem: Device 0000:0000 Capabilities: [100 v1] Advanced Error Reporting UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol- CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr- CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+ AERCap: First Error Pointer: 00, ECRCGenCap- ECRCGenEn- ECRCChkCap- ECRCChkEn- MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap- HeaderLog: 05000001 0000000f 02080002 00000000 Capabilities: [140 v1] Power Budgeting <?> Capabilities: [160 v1] Latency Tolerance Reporting Max snoop latency: 15728640ns Max no snoop latency: 15728640ns Capabilities: [180 v1] Multicast McastCap: MaxGroups 64, ECRCRegen- McastCtl: NumGroups 1, Enable- McastBAR: IndexPos 0, BaseAddr 0000000000000000 McastReceiveVec: 0000000000000000 McastBlockAllVec: 0000000000000000 McastBlockUntransVec: 0000000000000000 McastOverlayBAR: OverlaySize 0 (disabled), BaseAddr 0000000000000000 Capabilities: [1c0 v1] Secondary PCI Express LnkCtl3: LnkEquIntrruptEn- PerformEqu- LaneErrStat: 0 Capabilities: [200 v1] L1 PM Substates L1SubCap: PCI-PM_L1.2- PCI-PM_L1.1+ ASPM_L1.2- ASPM_L1.1+ L1_PM_Substates+ L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1- L1SubCtl2: Kernel driver in use: pcieport 02:00.0 PCI bridge: ASMedia Technology Inc. ASM2824 PCIe Gen3 Packet Switch (rev 01) (prog-if 00 [Normal decode]) Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0, Cache Line Size: 64 bytes Interrupt: pin A routed to IRQ 125 IOMMU group: 12 Bus: primary=02, secondary=03, subordinate=03, sec-latency=0 I/O behind bridge: [disabled] [32-bit] Memory behind bridge: 40700000-407fffff [32-bit] Prefetchable memory behind bridge: [disabled] [64-bit] Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR- BridgeCtl: Parity- SERR+ NoISA- VGA- VGA16+ MAbort- >Reset- FastB2B- PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn- Capabilities: [40] Power Management version 3 Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+) Status: D3 NoSoftRst- PME-Enable+ DSel=0 DScale=0 PME- Capabilities: [50] MSI: Enable+ Count=1/1 Maskable- 64bit+ Address: 00000000fee00298 Data: 0000 Capabilities: [80] Express (v2) Downstream Port (Slot+), MSI 00 DevCap: MaxPayload 512 bytes, PhantFunc 0 ExtTag+ RBE+ DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq- RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ MaxPayload 256 bytes, MaxReadReq 512 bytes DevSta: CorrErr+ NonFatalErr- FatalErr+ UnsupReq+ AuxPwr- TransPend- LnkCap: Port #0, Speed 8GT/s, Width x4, ASPM L1, Exit Latency L1 <64us ClockPM+ Surprise+ LLActRep+ BwNot+ ASPMOptComp+ LnkCtl: ASPM L1 Enabled; Disabled- CommClk+ ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 2.5GT/s, Width x1 TrErr- Train- SlotClk+ DLActive- BWMgmt+ ABWMgmt- SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise- Slot #0, PowerLimit 0W; Interlock- NoCompl- SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg- Control: AttnInd Unknown, PwrInd Unknown, Power- Interlock- SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet- Interlock- Changed: MRL- PresDet+ LinkState+ DevCap2: Completion Timeout: Not Supported, TimeoutDis- NROPrPrP- LTR+ 10BitTagComp- 10BitTagReq- OBFF Not Supported, ExtFmt- EETLPPrefix- EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit- FRS- ARIFwd+ AtomicOpsCap: Routing- DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR- 10BitTagReq- OBFF Disabled, ARIFwd- AtomicOpsCtl: EgressBlck- LnkCap2: Supported Link Speeds: 2.5-8GT/s, Crosslink- Retimer- 2Retimers- DRS- LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis+, Selectable De-emphasis: -3.5dB Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance Preset/De-emphasis: -6dB de-emphasis, 0dB preshoot LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete+ EqualizationPhase1+ EqualizationPhase2+ EqualizationPhase3+ LinkEqualizationRequest- Retimer- 2Retimers- CrosslinkRes: unsupported Capabilities: [c0] Subsystem: Device 0000:0000 Capabilities: [100 v1] Advanced Error Reporting UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol- UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol- CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+ CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+ AERCap: First Error Pointer: 14, ECRCGenCap- ECRCGenEn- ECRCChkCap- ECRCChkEn- MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap- HeaderLog: 04000001 0000000f 03000003 00000000 Capabilities: [140 v1] Power Budgeting <?> Capabilities: [160 v1] Latency Tolerance Reporting Max snoop latency: 0ns Max no snoop latency: 0ns Capabilities: [180 v1] Multicast McastCap: MaxGroups 64, ECRCRegen- McastCtl: NumGroups 1, Enable- McastBAR: IndexPos 0, BaseAddr 0000000000000000 McastReceiveVec: 0000000000000000 McastBlockAllVec: 0000000000000000 McastBlockUntransVec: 0000000000000000 McastOverlayBAR: OverlaySize 0 (disabled), BaseAddr 0000000000000000 Capabilities: [1c0 v1] Secondary PCI Express LnkCtl3: LnkEquIntrruptEn- PerformEqu- LaneErrStat: 0 Capabilities: [200 v1] L1 PM Substates L1SubCap: PCI-PM_L1.2- PCI-PM_L1.1+ ASPM_L1.2- ASPM_L1.1+ L1_PM_Substates+ L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1+ ASPM_L1.2- ASPM_L1.1+ L1SubCtl2: Kernel driver in use: pcieport I can post the rest of the SM2824 verbose output if it's helpful. I'm not a wizard at this, but to my knowledge the FatalErr+ would be much better as FatalErr- If anyone has any ideas or things to do let me know. Oh yeah, I've found some stuff like this online https://www.spinics.net/lists/linux-pci/msg140549.html But I'm too dumb to follow it. Maybe it's not of use.
July 27, 20232 yr Author I think I'm just going to give up. Is it possible if I go down to two drives that I don't lose any data? The pool was 4 2tb drives and I'll go to 2 2tb drives. Ideally I would get the pool back up, make sure nothing is missing/move it if it's there, then remake it so it's 2 mirrored drives.
July 31, 20232 yr Community Expert On 7/27/2023 at 2:15 PM, ausername11 said: Is it possible if I go down to two drives that I don't lose any data? Yes if it's btrfs, but you can only remove one device at a time, so it would need at least 3 working devices.
June 20, 20242 yr This same thing happened to me, BTW. Went from a Z490 board with this same QNAP card, 4x 2TB NVMes loaded to a Z690 board and now i have nothing but problems with it. It's seemingly hit and miss on whether the drives show up or not (depending on different combinations of drive population and onboard slot usage).
February 10Feb 10 On 6/20/2024 at 1:04 PM, Andiroo2 said:This same thing happened to me, BTW. Went from a Z490 board with this same QNAP card, 4x 2TB NVMes loaded to a Z690 board and now i have nothing but problems with it. It's seemingly hit and miss on whether the drives show up or not (depending on different combinations of drive population and onboard slot usage). Hi,i know this is late , a long time down the road. but in case anyone else hits this.the reason this is happening is because the ASM2824 controller sleeps the drives before the os has a chance to probe them (depends on nvme controller.) to fix this issue , you have to disable nvme sleep or change the time to wake, appending this to grub works : nvme_core.default_ps_max_latency_us=0 , you can try 500 to get a lower power state , but anything beyond that is likely to cause the disks to not show up at all.i have tested this on Z690 , Z790 and W680 chipsets , it solves the problem on both ASM2824 and PEX8729 , PEX88088 controllers.
February 10Feb 10 4 hours ago, MacModMachine said:you have to disable nvme sleep or change the time to wake, appending this to grub worksWhere do I add this? Is this in the flash drive config page, under Syslinux config?
February 10Feb 10 17 minutes ago, Andiroo2 said:Where do I add this? Is this in the flash drive config page, under Syslinux config?thats correct , syslinuxafter initrd=/bzroot should look like this : initrd=/bzroot nvme_core.default_ps_max_latency_us=0 , you can just add it to the end
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.