May 16, 20224 yr Recently had some issues so I rebuilt my system with a new LSI SAS9201-16 e card. It is running the p20 firmware. I'm getting a throughput average of around 40MB/sec. I'm wondering if this is normal speed seen by others with this card or is this due to a dual disk rebuild? Yep, I run dual parity and two drives seemed to have failed, so it is rebuilding those now. Diag attached, thanks! tower-diagnostics-20220516-0740.zip
May 16, 20224 yr Community Expert 35 minutes ago, Whaler_99 said: if this is normal speed Should be faster than that, I assume you're using the bottom slot, which is x4 only, still should give you around 100MB/s with 16 drives, check the current link speed/width with lspci -vv -s 05:00.0
May 16, 20224 yr Author It is in the second PCIe slot, so should be x8 on that board. Here are the results. 05:00.0 Serial Attached SCSI controller: Broadcom / LSI SAS2116 PCI-Express Fusion-MPT SAS-2 [Meteor] (rev 02) Subsystem: Broadcom / LSI 9201-16e 6Gb/s SAS/SATA PCIe x8 External HBA Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0, Cache Line Size: 64 bytes Interrupt: pin A routed to IRQ 29 IOMMU group: 12 Region 0: I/O ports at d000 [size=256] Region 1: Memory at d0300000 (64-bit, non-prefetchable) [size=16K] Region 3: Memory at d0100000 (64-bit, non-prefetchable) [size=256K] Expansion ROM at <ignored> [disabled] Capabilities: [50] Power Management version 3 Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-) Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME- Capabilities: [68] Express (v2) Endpoint, MSI 00 DevCap: MaxPayload 4096 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0.000W DevCtl: CorrErr+ NonFatalErr+ FatalErr+ UnsupReq+ RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset- MaxPayload 128 bytes, MaxReadReq 512 bytes DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend+ LnkCap: Port #0, Speed 5GT/s, Width x8, ASPM L0s, Exit Latency L0s <64ns ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp- LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+ ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 5GT/s (ok), Width x4 (downgraded) TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt- DevCap2: Completion Timeout: Range BC, TimeoutDis+ NROPrPrP- LTR- 10BitTagComp- 10BitTagReq- OBFF Not Supported, ExtFmt- EETLPPrefix- EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit- FRS- TPHComp- ExtTPHComp- AtomicOpsCap: 32bit- 64bit- 128bitCAS- DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR- OBFF Disabled, AtomicOpsCtl: ReqEn- LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis- Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete- EqualizationPhase1- EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest- Retimer- 2Retimers- CrosslinkRes: unsupported Capabilities: [d0] Vital Product Data pcilib: sysfs_read_vpd: read failed: Input/output error Not readable Capabilities: [a8] MSI: Enable- Count=1/1 Maskable- 64bit+ Address: 0000000000000000 Data: 0000 Capabilities: [c0] MSI-X: Enable+ Count=15 Masked- Vector table: BAR=1 offset=00002000 PBA: BAR=1 offset=00003800 Capabilities: [100 v1] Advanced Error Reporting UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol- CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr- CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+ AERCap: First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn- MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap- HeaderLog: 00000000 00000000 00000000 00000000 Capabilities: [138 v1] Power Budgeting <?> Capabilities: [150 v1] Single Root I/O Virtualization (SR-IOV) IOVCap: Migration-, Interrupt Message Number: 000 IOVCtl: Enable- Migration- Interrupt- MSE- ARIHierarchy- IOVSta: Migration- Initial VFs: 7, Total VFs: 7, Number of VFs: 0, Function Dependency Link: 00 VF offset: 1, stride: 1, Device ID: 0064 Supported Page Size: 00000553, System Page Size: 00000001 Region 0: Memory at 00000000d0304000 (64-bit, non-prefetchable) Region 2: Memory at 00000000d0140000 (64-bit, non-prefetchable) VF Migration: offset: 00000000, BIR: 0 Capabilities: [190 v1] Alternative Routing-ID Interpretation (ARI) ARICap: MFVC- ACS-, Next Function: 0 ARICtl: MFVC- ACS-, Function Group: 0 Kernel driver in use: mpt3sas Kernel modules: mpt3sas
May 16, 20224 yr Community Expert 10 minutes ago, Whaler_99 said: so should be x8 on that board It's x4 only, so link is correct: 11 minutes ago, Whaler_99 said: Speed 5GT/s (ok), Width x4 (downgraded)
May 16, 20224 yr Author <sigh> frigging mobo manufacturers... checked manual, sure enough, then second x16 slot is limited to @x4 speeds... like why bother, lol? Looks like I might try and find another motherboard with two slots running at x8... Thanks for your help @JorgeB
May 16, 20224 yr Community Expert 8 minutes ago, Whaler_99 said: with two slots running at x8..@ It might help but like mentioned even @ x4 I would expect around 100MB/s if that was the limiting factor.
May 16, 20224 yr Author Yes, I previously had dual LSI 8 port internal cards, which when doing a parity check, I'd get 100Mb/s.... So, dug a little deeper, looks like I ordered the wrong cables. https://www.amazon.ca/gp/product/B013G4EX9K/ref=ppx_yo_dt_b_asin_title_o01_s01?ie=UTF8&psc=1 SAS1 cables... not SAS2....
May 16, 20224 yr Community Expert That won't be the problem, SAS1 = SATA2, so still 300MB/s per drive, also checked a couple of disks and they are linking @ SATA3
May 16, 20224 yr Author Huh... seems to be some sort of bandwidth block, the estimated speed never goes above the 40'ish MB/s mark... The only thing I can think of, since it's the LSI card and cables I changed, something is up with the settings on the card itself. Although when going through it, there isn't a lot of options even on the card to set.
May 16, 20224 yr Community Expert Try a test with the diskspeed docker, parity check must not be ongoing to get correct results, not sure if it still works if you pause it.
May 18, 20224 yr Author running disk speed tests now, shows all signaling at 6Gb on all drives. Curiously, I noticed this in the log: May 16 07:45:00 Tower root: Fix Common Problems: Warning: Docker Application binhex-plex has an update available for it May 16 07:45:00 Tower root: Fix Common Problems: Warning: Docker Application duckdns has an update available for it May 16 07:45:04 Tower root: Fix Common Problems: Warning: preclear.disk.plg Not Compatible with Unraid version 6.9.2 May 16 07:45:07 Tower root: Fix Common Problems: Warning: Write Cache is disabled on disk2 May 16 07:45:07 Tower root: Fix Common Problems: Warning: Write Cache is disabled on disk3 May 16 07:45:07 Tower root: Fix Common Problems: Warning: Write Cache is disabled on disk4 May 16 07:45:07 Tower root: Fix Common Problems: Warning: Write Cache is disabled on disk5 May 16 07:45:07 Tower root: Fix Common Problems: Warning: Write Cache is disabled on disk6 May 16 07:45:07 Tower root: Fix Common Problems: Warning: Write Cache is disabled on disk7 May 16 07:45:07 Tower root: Fix Common Problems: Warning: Write Cache is disabled on disk8 May 16 07:45:07 Tower root: Fix Common Problems: Warning: Write Cache is disabled on disk9 May 16 07:45:07 Tower root: Fix Common Problems: Warning: Write Cache is disabled on disk10 May 16 07:45:07 Tower root: Fix Common Problems: Warning: Write Cache is disabled on disk11 May 16 07:45:07 Tower root: Fix Common Problems: Warning: Write Cache is disabled on disk12 May 16 07:45:07 Tower root: Fix Common Problems: Warning: Write Cache is disabled on disk13 May 16 07:45:07 Tower root: Fix Common Problems: Warning: Write Cache is disabled on disk14 May 16 07:45:07 Tower root: Fix Common Problems: Warning: Write Cache is disabled on disk15 May 16 07:45:07 Tower root: Fix Common Problems: Warning: Write Cache is disabled on disk16 May 16 07:45:07 Tower root: Fix Common Problems: Warning: Write Cache is disabled on disk17 May 16 07:45:08 Tower root: Fix Common Problems: Other Warning: Background notifications not enabled This is coming up since I swapped to this new controller and three disks are not listed, the three on the internal controller. I wonder if not have the cache enabled on all these disks could be impacting the throughput.
May 18, 20224 yr Community Expert 1 minute ago, Whaler_99 said: I wonder if not have the cache enabled on all these disks could be impacting the throughput. Yes, since it's disable on the disks being rebuilt, it won't affect read speed.
May 18, 20224 yr Author Some weird numbers... Doing same test on the internal controller shows single and multi tests almost identical... Edited May 18, 20224 yr by Whaler_99
May 18, 20224 yr Community Expert Not very weird since a bottleneck is expected, due to being in an x4 slot.
May 18, 20224 yr Author Going to shut down later today when home from work, swap the controller with the GPU, as the GPU doesn't need x16 throughput and then test and see what happens. I also found the post on enabling the write cache, so will work on that as well.
May 19, 20224 yr Author Fixed things up, enabled write cache on the disks, moved the controller into the top slot, much better looking now.
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.