Jump to content

Slow throughput during rebuild/parity - LSI SAS 9201


Recommended Posts

Recently had some issues so I rebuilt my system with a new LSI SAS9201-16 e card. It is running the p20 firmware. I'm getting a throughput average of around 40MB/sec. I'm wondering if this is normal speed seen by others with this card or is this due to a dual disk rebuild? Yep, I run dual parity and two drives seemed to have failed, so it is rebuilding those now. Diag attached, thanks!

 

image.thumb.png.fd5cb0521387af6d9706e3e971a5fbab.png

tower-diagnostics-20220516-0740.zip

Link to comment

It is in the second PCIe slot, so should be x8 on that board.  Here are the results.

 

05:00.0 Serial Attached SCSI controller: Broadcom / LSI SAS2116 PCI-Express Fusion-MPT SAS-2 [Meteor] (rev 02)
        Subsystem: Broadcom / LSI 9201-16e 6Gb/s SAS/SATA PCIe x8 External HBA
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0, Cache Line Size: 64 bytes
        Interrupt: pin A routed to IRQ 29
        IOMMU group: 12
        Region 0: I/O ports at d000 [size=256]
        Region 1: Memory at d0300000 (64-bit, non-prefetchable) [size=16K]
        Region 3: Memory at d0100000 (64-bit, non-prefetchable) [size=256K]
        Expansion ROM at <ignored> [disabled]
        Capabilities: [50] Power Management version 3
                Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
                Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [68] Express (v2) Endpoint, MSI 00
                DevCap: MaxPayload 4096 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
                        ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0.000W
                DevCtl: CorrErr+ NonFatalErr+ FatalErr+ UnsupReq+
                        RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset-
                        MaxPayload 128 bytes, MaxReadReq 512 bytes
                DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend+
                LnkCap: Port #0, Speed 5GT/s, Width x8, ASPM L0s, Exit Latency L0s <64ns
                        ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
                LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 5GT/s (ok), Width x4 (downgraded)
                        TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
                DevCap2: Completion Timeout: Range BC, TimeoutDis+ NROPrPrP- LTR-
                         10BitTagComp- 10BitTagReq- OBFF Not Supported, ExtFmt- EETLPPrefix-
                         EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
                         FRS- TPHComp- ExtTPHComp-
                         AtomicOpsCap: 32bit- 64bit- 128bitCAS-
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR- OBFF Disabled,
                         AtomicOpsCtl: ReqEn-
                LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
                         Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
                         Compliance De-emphasis: -6dB
                LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete- EqualizationPhase1-
                         EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest-
                         Retimer- 2Retimers- CrosslinkRes: unsupported
        Capabilities: [d0] Vital Product Data
pcilib: sysfs_read_vpd: read failed: Input/output error
                Not readable
        Capabilities: [a8] MSI: Enable- Count=1/1 Maskable- 64bit+
                Address: 0000000000000000  Data: 0000
        Capabilities: [c0] MSI-X: Enable+ Count=15 Masked-
                Vector table: BAR=1 offset=00002000
                PBA: BAR=1 offset=00003800
        Capabilities: [100 v1] Advanced Error Reporting
                UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
                CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr-
                CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
                AERCap: First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
                        MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
                HeaderLog: 00000000 00000000 00000000 00000000
        Capabilities: [138 v1] Power Budgeting <?>
        Capabilities: [150 v1] Single Root I/O Virtualization (SR-IOV)
                IOVCap: Migration-, Interrupt Message Number: 000
                IOVCtl: Enable- Migration- Interrupt- MSE- ARIHierarchy-
                IOVSta: Migration-
                Initial VFs: 7, Total VFs: 7, Number of VFs: 0, Function Dependency Link: 00
                VF offset: 1, stride: 1, Device ID: 0064
                Supported Page Size: 00000553, System Page Size: 00000001
                Region 0: Memory at 00000000d0304000 (64-bit, non-prefetchable)
                Region 2: Memory at 00000000d0140000 (64-bit, non-prefetchable)
                VF Migration: offset: 00000000, BIR: 0
        Capabilities: [190 v1] Alternative Routing-ID Interpretation (ARI)
                ARICap: MFVC- ACS-, Next Function: 0
                ARICtl: MFVC- ACS-, Function Group: 0
        Kernel driver in use: mpt3sas
        Kernel modules: mpt3sas

 

Link to comment

Huh... seems to be some sort of bandwidth block, the estimated speed never goes above the 40'ish MB/s mark...

The only thing I can think of, since it's the LSI card and cables I changed, something is up with the settings on the card itself. Although when going through it, there isn't a lot of options even on the card to set.

Link to comment

running disk speed tests now, shows all signaling at 6Gb on all drives. Curiously, I noticed this in the log:

 

May 16 07:45:00 Tower root: Fix Common Problems: Warning: Docker Application binhex-plex has an update available for it
May 16 07:45:00 Tower root: Fix Common Problems: Warning: Docker Application duckdns has an update available for it
May 16 07:45:04 Tower root: Fix Common Problems: Warning: preclear.disk.plg Not Compatible with Unraid version 6.9.2
May 16 07:45:07 Tower root: Fix Common Problems: Warning: Write Cache is disabled on disk2
May 16 07:45:07 Tower root: Fix Common Problems: Warning: Write Cache is disabled on disk3
May 16 07:45:07 Tower root: Fix Common Problems: Warning: Write Cache is disabled on disk4
May 16 07:45:07 Tower root: Fix Common Problems: Warning: Write Cache is disabled on disk5
May 16 07:45:07 Tower root: Fix Common Problems: Warning: Write Cache is disabled on disk6
May 16 07:45:07 Tower root: Fix Common Problems: Warning: Write Cache is disabled on disk7
May 16 07:45:07 Tower root: Fix Common Problems: Warning: Write Cache is disabled on disk8
May 16 07:45:07 Tower root: Fix Common Problems: Warning: Write Cache is disabled on disk9
May 16 07:45:07 Tower root: Fix Common Problems: Warning: Write Cache is disabled on disk10
May 16 07:45:07 Tower root: Fix Common Problems: Warning: Write Cache is disabled on disk11
May 16 07:45:07 Tower root: Fix Common Problems: Warning: Write Cache is disabled on disk12
May 16 07:45:07 Tower root: Fix Common Problems: Warning: Write Cache is disabled on disk13
May 16 07:45:07 Tower root: Fix Common Problems: Warning: Write Cache is disabled on disk14
May 16 07:45:07 Tower root: Fix Common Problems: Warning: Write Cache is disabled on disk15
May 16 07:45:07 Tower root: Fix Common Problems: Warning: Write Cache is disabled on disk16
May 16 07:45:07 Tower root: Fix Common Problems: Warning: Write Cache is disabled on disk17
May 16 07:45:08 Tower root: Fix Common Problems: Other Warning: Background notifications not enabled

 

This is coming up since I swapped to this new controller and three disks are not listed, the three on the internal controller. I wonder if not have the cache enabled on all these disks could be impacting the throughput. 

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

×
×
  • Create New...