Replaced disk rebuild speed query

[BOGLOAD]

Hi all.

 

I've recently decided to replace some ageing HDDs (some with many reallocated sectors, and some just smaller/slower drives). I've noticed that each time I replace a disk (or two max at a time), the parity rebuild read speed tops out at a collective 1.5GB/s.

 

My current setup includes an LSI 9211-8i HBA connected to and a HP SAS Expander using both channels. Build is inside a Norco 4224 case, 5600x CPU, 32gb RAM, Gigabyte X570 UD board with 750W PSU.

 

Research shows that  SATA 6.0GB/s capable drives operate @ 3Gbps as a result of the HP SAS Expander's limitations, and this is consistent with what I'm seeing in the Disk Identity tab for all drives (i.e.: SATA 3.3, 6.0 Gb/s (current: 3.0 Gb/s)). All drives currently are 8tb <-> 12tb IronWolf flavour. I previously had 2x 8tb WD REDs that could only sync @ 1.5Gb/s, so I replaced them as I suspected it was a firmware issue of sorts and just couldn't be bothered diagnosing why.

 

During a rebuild, these are the speeds I'm seeing:

 

 

My question is, is this collective read speed normal and/or expected? Every time I've done this, read has collectively topped out @ 1.5GB/s. A rebuild of a 10tb drive is currently estimated to take 1 day, 7 hours (initial estimate). Similarly, write speeds are about 85-90MB/s for each drive I replace (160-180MB/s for 2 drives, still 1.5GB/s read speed). I've just got this nagging feeling that the current hardware setup may be limiting the read speeds somehow, possibly the write a little bit also (I've seen some say parity rebuild as fast as ~120 MB/s).
 

FYI: I was considering replacing the HBA+SAS card with a single, 24 port HBA (something with 6x backplane support) to reduce cable clutter and improve speeds a little, but they're rather expensive right now.

 

Any insight/commentary would be greatly appreciated, cheers

 

 

[JorgeB]

31 minutes ago, BOGLOAD said:

My current setup includes an LSI 9211-8i HBA connected to and a HP SAS Expander using both channels.

This should give 2.2GB/s, check the LSI link speed, post the output of:

 

lspci -d 1000: -vv

 

 

[BOGLOAD]

17 minutes ago, JorgeB said:

lspci -d 1000: -vv

Output here:

 

root@Tower:~# lspci -d 1000: -vv
03:00.0 Serial Attached SCSI controller: Broadcom / LSI SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] (rev 02)
        Subsystem: Broadcom / LSI 9210-8i
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0, Cache Line Size: 64 bytes
        Interrupt: pin A routed to IRQ 38
        IOMMU group: 20
        Region 0: I/O ports at e000 [size=256]
        Region 1: Memory at c0440000 (64-bit, non-prefetchable) [size=16K]
        Region 3: Memory at c0000000 (64-bit, non-prefetchable) [size=256K]
        Expansion ROM at <ignored> [disabled]
        Capabilities: [50] Power Management version 3
                Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
                Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [68] Express (v2) Endpoint, MSI 00
                DevCap: MaxPayload 4096 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
                        ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0.000W
                DevCtl: CorrErr+ NonFatalErr+ FatalErr+ UnsupReq+
                        RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset-
                        MaxPayload 128 bytes, MaxReadReq 512 bytes
                DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend+
                LnkCap: Port #0, Speed 5GT/s, Width x8, ASPM L0s, Exit Latency L0s <64ns
                        ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
                LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 5GT/s (ok), Width x4 (downgraded)
                        TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
                DevCap2: Completion Timeout: Range BC, TimeoutDis+ NROPrPrP- LTR-
                         10BitTagComp- 10BitTagReq- OBFF Not Supported, ExtFmt- EETLPPrefix-
                         EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
                         FRS- TPHComp- ExtTPHComp-
                         AtomicOpsCap: 32bit- 64bit- 128bitCAS-
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR- OBFF Disabled,
                         AtomicOpsCtl: ReqEn-
                LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
                         Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
                         Compliance De-emphasis: -6dB
                LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete- EqualizationPhase1-
                         EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest-
                         Retimer- 2Retimers- CrosslinkRes: unsupported
        Capabilities: [d0] Vital Product Data
pcilib: sysfs_read_vpd: read failed: No such device
                Not readable
        Capabilities: [a8] MSI: Enable- Count=1/1 Maskable- 64bit+
                Address: 0000000000000000  Data: 0000
        Capabilities: [c0] MSI-X: Enable+ Count=15 Masked-
                Vector table: BAR=1 offset=00002000
                PBA: BAR=1 offset=00003800
        Capabilities: [100 v1] Advanced Error Reporting
                UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
                CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr-
                CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
                AERCap: First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
                        MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
                HeaderLog: 00000000 00000000 00000000 00000000
        Capabilities: [138 v1] Power Budgeting <?>
        Capabilities: [150 v1] Single Root I/O Virtualization (SR-IOV)
                IOVCap: Migration-, Interrupt Message Number: 000
                IOVCtl: Enable- Migration- Interrupt- MSE- ARIHierarchy-
                IOVSta: Migration-
                Initial VFs: 16, Total VFs: 16, Number of VFs: 0, Function Dependency Link: 00
                VF offset: 1, stride: 1, Device ID: 0072
                Supported Page Size: 00000553, System Page Size: 00000001
                Region 0: Memory at 00000000c0444000 (64-bit, non-prefetchable)
                Region 2: Memory at 00000000c0040000 (64-bit, non-prefetchable)
                VF Migration: offset: 00000000, BIR: 0
        Capabilities: [190 v1] Alternative Routing-ID Interpretation (ARI)
                ARICap: MFVC- ACS-, Next Function: 0
                ARICtl: MFVC- ACS-, Function Group: 0
        Kernel driver in use: mpt3sas
        Kernel modules: mpt3sas

root@Tower:~# 

 

[JorgeB]

2 minutes ago, BOGLOAD said:

LnkSta: Speed 5GT/s (ok), Width x4 (downgraded)

It's only linking at x4, than means at theoretical max speed of 2GB/s, about 75% of that is usable, so 1.5GB/s seems about right.

[BOGLOAD]

1 minute ago, JorgeB said:

It's only linking at x4, than means at theoretical max speed of 2GB/s, about 75% of that is usable, so 1.5GB/s seems about right.

That would probably be it. Thanks @JorgeB. Turns out my motherboard drops from 16x on the PCIE main slot when more devices are added down to 8x/4x/4x (my GPU is sitting in the 8x slot). This post here suggests the behaviour: 

 

 

Shame, might have to replace the actual motherboard to get an improvement.