October 13, 20241 yr I just swapped one of my 18tb parity drives for a 20tb and damn, it took a long time...almost 3 days. When i was looking at the history, it looks like around the time that i swapped from a 9211-8i to a 9207-8i with a RES2V240 sas expander my speeds dropped quite a bit. Those red strike outs were 16tb disk rebuilds due to a bad connection, those rebuild speeds were 76-83MB/s. figured i'd look into it before i swap the second parity for a 20tb and start the rebuild. I'm using dual link cables between the hba and the expander card and sas to sata breakout cables on the other 4 ports of the expander. Is this expected speeds or is something that i can look into. The server isn't doing a crazy amount of work during this last rebuild of th 20tb drive, normal arr's and plex. I did have the syslog server running since i was having an issue, but its now fixed (fingers crossed). One last thing, when i did the parity rebuild with the new 20tb drive it said "Data Rebuild Complete (0 errors) and next to it under Description it said "canceled", is that normal? I didn't cancel anything so it should have finished just fine. On the Main tab both parity drives are green. tower-diagnostics-20241013-0852.zip
October 14, 20241 yr Author 2 hours ago, JorgeB said: Post the output from: lspci -d 1000: -vv I've got the second parity rebuild going....still do this or wait?
October 14, 20241 yr Author 14 hours ago, JorgeB said: Post the output from: lspci -d 1000: -vv 01:00.0 Serial Attached SCSI controller: Broadcom / LSI SAS2308 PCI-Express Fusion-MPT SAS-2 (rev 05) Subsystem: Super Micro Computer Inc Onboard SAS2308 PCI-Express Fusion-MPT SAS-2 Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0, Cache Line Size: 64 bytes Interrupt: pin A routed to IRQ 16 IOMMU group: 16 Region 0: I/O ports at 4000 Region 1: Memory at 80b40000 (64-bit, non-prefetchable) Region 3: Memory at 80b00000 (64-bit, non-prefetchable) Expansion ROM at 80a00000 [disabled] Capabilities: [50] Power Management version 3 Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-) Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME- Capabilities: [68] Express (v2) Endpoint, MSI 00 DevCap: MaxPayload 4096 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0W DevCtl: CorrErr+ NonFatalErr+ FatalErr+ UnsupReq+ RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset- MaxPayload 256 bytes, MaxReadReq 512 bytes DevSta: CorrErr+ NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend- LnkCap: Port #0, Speed 8GT/s, Width x8, ASPM L0s, Exit Latency L0s <64ns ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+ LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+ ExtSynch+ ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 8GT/s, Width x4 (downgraded) TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt- DevCap2: Completion Timeout: Range BC, TimeoutDis+ NROPrPrP- LTR- 10BitTagComp- 10BitTagReq- OBFF Not Supported, ExtFmt- EETLPPrefix- EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit- FRS- TPHComp- ExtTPHComp- AtomicOpsCap: 32bit- 64bit- 128bitCAS- DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR- 10BitTagReq- OBFF Disabled, AtomicOpsCtl: ReqEn- LnkCap2: Supported Link Speeds: 2.5-8GT/s, Crosslink- Retimer- 2Retimers- DRS- LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis- Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance Preset/De-emphasis: -6dB de-emphasis, 0dB preshoot LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete+ EqualizationPhase1+ EqualizationPhase2+ EqualizationPhase3+ LinkEqualizationRequest- Retimer- 2Retimers- CrosslinkRes: unsupported Capabilities: [d0] Vital Product Data pcilib: sysfs_read_vpd: read failed: No such device Not readable Capabilities: [a8] MSI: Enable- Count=1/1 Maskable- 64bit+ Address: 0000000000000000 Data: 0000 Capabilities: [c0] MSI-X: Enable+ Count=16 Masked- Vector table: BAR=1 offset=0000e000 PBA: BAR=1 offset=0000f000 Capabilities: [100 v2] Advanced Error Reporting UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol- CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr- CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+ AERCap: First Error Pointer: 00, ECRCGenCap- ECRCGenEn- ECRCChkCap- ECRCChkEn- MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap- HeaderLog: 00000000 00000000 00000000 00000000 Capabilities: [1e0 v1] Secondary PCI Express LnkCtl3: LnkEquIntrruptEn- PerformEqu- LaneErrStat: 0 Capabilities: [1c0 v1] Power Budgeting <?> Capabilities: [190 v1] Dynamic Power Allocation <?> Capabilities: [148 v1] Alternative Routing-ID Interpretation (ARI) ARICap: MFVC- ACS-, Next Function: 0 ARICtl: MFVC- ACS-, Function Group: 0 Kernel driver in use: mpt3sas Kernel modules: mpt3sas
October 14, 20241 yr Community Expert 11 minutes ago, danimal86 said: LnkSta: Speed 8GT/s, Width x4 (downgraded) Link is downgraded, but just that by itself doesn't justify the speed you are seeing, post the diags saved during the next parity check.
October 14, 20241 yr Author 10 minutes ago, JorgeB said: Link is downgraded, but just that by itself doesn't justify the speed you are seeing, post the diags saved during the next parity check. Parity rebuild fine (attached)? or wait till during a check tower-diagnostics-20241014-0558.zip
October 14, 20241 yr Community Expert Log is files with device errors, for multiple disks, these are usually a power/connection problem, and would explain the lower speeds.
October 14, 20241 yr Author 1 hour ago, JorgeB said: Log is files with device errors, for multiple disks, these are usually a power/connection problem, and would explain the lower speeds. Dang. I just swapped to a new PSU, and hba cables. I have backup hba card SAS expander card. I'll swap it once the rebuild is done.
October 17, 20241 yr Author On 10/14/2024 at 7:22 AM, JorgeB said: Log is files with device errors, for multiple disks, these are usually a power/connection problem, and would explain the lower speeds. I just swapped to a different hba card. I've attached new diagnostics to see if there are still disk errors. For some reason upon startup, it started a parity check, so for some reason it didn't do a clean shutdown. I'll probably bump up the Shutdown Time-Out to 120sec I re-ran the command and here's the output with the new hba. Looks like linksta is Speed 8GT/s, Width x8 before it was x4 root@Tower:~# lspci -d 1000: -vv 01:00.0 Serial Attached SCSI controller: Broadcom / LSI SAS2308 PCI-Express Fusion-MPT SAS-2 (rev 05) Subsystem: Broadcom / LSI SAS2308 PCI-Express Fusion-MPT SAS-2 Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0, Cache Line Size: 64 bytes Interrupt: pin A routed to IRQ 16 IOMMU group: 16 Region 0: I/O ports at 4000 [disabled] Region 1: Memory at 80b40000 (64-bit, non-prefetchable) Region 3: Memory at 80b00000 (64-bit, non-prefetchable) Expansion ROM at 80a00000 [disabled] Capabilities: [50] Power Management version 3 Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-) Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME- Capabilities: [68] Express (v2) Endpoint, MSI 00 DevCap: MaxPayload 4096 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0W DevCtl: CorrErr+ NonFatalErr+ FatalErr+ UnsupReq+ RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset- MaxPayload 256 bytes, MaxReadReq 512 bytes DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend- LnkCap: Port #0, Speed 8GT/s, Width x8, ASPM L0s, Exit Latency L0s <64ns ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+ LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+ ExtSynch+ ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 8GT/s, Width x8 TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt- DevCap2: Completion Timeout: Range BC, TimeoutDis+ NROPrPrP- LTR- 10BitTagComp- 10BitTagReq- OBFF Not Supported, ExtFmt- EETLPPrefix- EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit- FRS- TPHComp- ExtTPHComp- AtomicOpsCap: 32bit- 64bit- 128bitCAS- DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR- 10BitTagReq- OBFF Disabled, AtomicOpsCtl: ReqEn- LnkCap2: Supported Link Speeds: 2.5-8GT/s, Crosslink- Retimer- 2Retimers- DRS- LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis- Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance Preset/De-emphasis: -6dB de-emphasis, 0dB preshoot LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete+ EqualizationPhase1+ EqualizationPhase2+ EqualizationPhase3+ LinkEqualizationRequest- Retimer- 2Retimers- CrosslinkRes: unsupported Capabilities: [d0] Vital Product Data pcilib: sysfs_read_vpd: read failed: No such device Not readable Capabilities: [a8] MSI: Enable- Count=1/1 Maskable- 64bit+ Address: 0000000000000000 Data: 0000 Capabilities: [c0] MSI-X: Enable+ Count=16 Masked- Vector table: BAR=1 offset=0000e000 PBA: BAR=1 offset=0000f000 Capabilities: [100 v2] Advanced Error Reporting UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol- CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr- CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+ AERCap: First Error Pointer: 00, ECRCGenCap- ECRCGenEn- ECRCChkCap- ECRCChkEn- MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap- HeaderLog: 00000000 00000000 00000000 00000000 Capabilities: [1e0 v1] Secondary PCI Express LnkCtl3: LnkEquIntrruptEn- PerformEqu- LaneErrStat: 0 Capabilities: [1c0 v1] Power Budgeting <?> Capabilities: [190 v1] Dynamic Power Allocation <?> Capabilities: [148 v1] Alternative Routing-ID Interpretation (ARI) ARICap: MFVC- ACS-, Next Function: 0 ARICtl: MFVC- ACS-, Function Group: 0 Kernel driver in use: mpt3sas Kernel modules: mpt3sas tower-diagnostics-20241017-0742.zip
October 17, 20241 yr Community Expert 24 minutes ago, danimal86 said: Looks like linksta is Speed 8GT/s, Width x8 Yep, and no errors so far.
October 17, 20241 yr Author 17 minutes ago, JorgeB said: Yep, and no errors so far. Woohoo! I'm happy to hopefully have my speeds back. Anything else i can do to further check it, maybe a parity check or possibly running the Disk Speeds docker test?
October 17, 20241 yr Community Expert A parity check should be a good test, especially for the disk/controller errors.
October 17, 20241 yr Author 51 minutes ago, JorgeB said: A parity check should be a good test, especially for the disk/controller errors. Ok, i'll run one now. I'm trying to figure out why something is reading/writing to my last drive or 2 and the parity about every 3 seconds. I thought it bazarr looking for subtitles since there are so many .srt files in the disk activity but i stopped the bazaar docker and its still doing the same thing of read/write to both parity and disk 11 and File Activity (shares activity and disk activity) is just blank after i cleared it. Got any tricks to figure out what the hell keeps writing to disk 10/11 and parity drives? i'm sure a constant read/write isn't helping my speeds any. I attached an updated diag in case it would help tower-diagnostics-20241017-1031.zip
October 17, 20241 yr Community Expert Try stopping the docker service, if it helps, then stat one container at a time.
October 17, 20241 yr Author On 10/17/2024 at 10:48 AM, JorgeB said: Try stopping the docker service, if it helps, then stat one container at a time. So i just stopped all the dockers one by one and it might be tautulli...it definitely helped with the read/write of both parity and disk 11 but didn't fix it completly. I'm still seeing the read/write but its like a few minutes apart instead of every 3 seconds. still not enough to let the drives spin down after 15min One thing i'm noticing is that when i start/stop all dockers, its pretty dang slow. I dont remember it taking this long to start/stop 9-10 dockers. Edit: i just started up all my normal dockers, except tautulli and its back to reads/writes every second or so. Frustrating! Edit2: I figured it out, i had the docker image on the array for some reason. I moved it to the cache and it seems like its no longer spinning up disks. Edited October 22, 20241 yr by danimal86
October 18, 20241 yr Author 22 hours ago, JorgeB said: A parity check should be a good test, especially for the disk/controller errors. So i let the parity check go overnight, hoping it would be faster than it was before but it seems like my issue is not fixed. Over the 14.5hrs its been running, its checked 4.21tb.....so around a speed of 60 MB/s I've gotten 3 crc errors on Parity 1 drive (sdc), i hadn't had that happen before. I believe that i have the two parity drives plugged into the mobo not through the hba. I'm sure the diag shows disk errors, are there errors on disks other than sdc? I'm thinking i'll go ahead and stop the parity check and try reseating the cables to sdc and switching the port on the mobo. think thats a good idea? tower-diagnostics-20241018-0731.zip
October 18, 20241 yr Author Just now, JorgeB said: Constant errors for disk2, check/replace cables. Is it a bad idea to pause parity check, shutdown, fix cables, restart and resume check does it even work like that or do i just need to cancel the check, fix it and then restart the check
October 18, 20241 yr Community Expert 1 minute ago, danimal86 said: Is it a bad idea to pause parity check, shutdown, fix cables, restart and resume check The parity check would restart again from the beginning.
October 18, 20241 yr Author 6 minutes ago, itimpi said: The parity check would restart again from the beginning. ok, i'm fine with that. I'll just cancel it and try to fix the cables and restart it.
October 18, 20241 yr Author I used brand new Sabrent sata cables for parity 1 and disk 2 and plugged them directly into the motherboard sata ports. Parity check didn't start automagically so i started a new one. I'll let it run for a while an post up a new diagnostic.
October 21, 20241 yr Author On 10/18/2024 at 7:41 AM, JorgeB said: Constant errors for disk2, check/replace cables. Parity check completed, average speed of 100MB/s, not quite the 150 i was getting earlier, but it was also a faster drive i believe ( 18tb red pro) vs white 20tb Server still feels a little slow, it took about 60second to process that diagnostic. Hopefully no more disk errors. tower-diagnostics-20241020-1745.zip
October 21, 20241 yr Author 6 hours ago, JorgeB said: Still some errors, for disk1 now. I'm not sure what else to do. I put in a new PSU, disk1 is now plugged directly into the mobo with a brand new sata cable...... The only thing I can think of is that I believe it's a white label drive that I have kapton tape over the 3.3v pin...I figure that if that wasn't working the drive wouldn't be visible. Maybe I should clip the 3.3v wire on the PSU sata cable instead. I could rebuild it with a red pro I have also,
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.