Vr2Io Posted January 8, 2020 Share Posted January 8, 2020 (edited) 1 hour ago, icemansid said: old drives connected to this controller will cause drive speed issues across all of the drives. I don't think so, but you can troubleshoot in this direction. As mention before, HBA should have PCIe switch and your mainboard also have PCIe switch, those were in cascade and may have compatible problem. From Syslog, this have a sign, LSI HBA only running in PCIe Gen 1 speed, but bandwidth still have 16Gb/s. ( Even all problem solve, throughput will ceiling at this too ) Dec 30 20:01:55 UnRAID-Backup kernel: pci 0000:08:00.0: 16.000 Gb/s available PCIe bandwidth, limited by 2.5 GT/s x8 link at 0000:00:01.0 (capable of 63.008 Gb/s with 8 GT/s x8 link) I also notice you have P420i HBA, 1 or 2 disk connect, could you unplug it and try. Which Expander you got ?? Edited January 8, 2020 by Benson Quote Link to comment
icemansid Posted January 8, 2020 Author Share Posted January 8, 2020 I am using the HP 468405-002 and using a PCI-e to USB3 adpater to power it. The p420i is the onboard running the HP OEM dirves in a raid array acting as a cache drive. Ive rebooted and installed a second SAS2308 card for testing as well. Ive also attached a fresh diagnostic report. I dont exactly follow what you mean by PCIe switch. unraid-backup-diagnostics-20200108-1436.zip Quote Link to comment
Vr2Io Posted January 8, 2020 Share Posted January 8, 2020 (edited) From new Syslog, the second SAS2308 card also run at PCIe Gen 1 speed. In fact all device were in Gen 1, but machine support PCIe Gen 3. Does you wrongly set the limit in BIOS ? 25 minutes ago, icemansid said: I dont exactly follow what you mean by PCIe switch. It like expander to provide a function to let you connect more device but share same bandwidth, just expander service for SAS device. How about the result on 2nd 2308 HBA ?? Edited January 8, 2020 by Benson Quote Link to comment
icemansid Posted January 9, 2020 Author Share Posted January 9, 2020 3 hours ago, Benson said: From new Syslog, the second SAS2308 card also run at PCIe Gen 1 speed. In fact all device were in Gen 1, but machine support PCIe Gen 3. Does you wrongly set the limit in BIOS ? It like expander to provide a function to let you connect more device but share same bandwidth, just expander service for SAS device. How about the result on 2nd 2308 HBA ?? 2nd 2308 gives very similar results at this point. Nothing in the BIOS stands out but I have an HPE storage engineer looking at it as well. Hopefully he can assist with the PCI negotiations.. Quote Link to comment
JorgeB Posted January 9, 2020 Share Posted January 9, 2020 11 hours ago, Benson said: Dec 30 20:01:55 UnRAID-Backup kernel: pci 0000:08:00.0: 16.000 Gb/s available PCIe bandwidth, limited by 2.5 GT/s x8 link at 0000:00:01.0 (capable of 63.008 Gb/s with 8 GT/s x8 link) I didn't even look at the diags for this since diskspeed docker should report the correct link speed and it was reporting PCIe 3.0, @jbartlettmight want to take a look at this. Quote Link to comment
jbartlett Posted January 9, 2020 Share Posted January 9, 2020 The DiskSpeed app simply reports what the dd command is outputting. If you're having issues reading all the drives at once, try removing half of the drives and then run a controller benchmark to see if you get the same results or better. If the same, remove the existing and hook the others back up. If better, then add a couple drives and do another benchmark. Rinse & repeat. See if there's a magic number or a given drive that's causing the issue. Quote Link to comment
JorgeB Posted January 9, 2020 Share Posted January 9, 2020 1 hour ago, jbartlett said: The DiskSpeed app simply reports what the dd command is outputting. I meant the link speed: 13 hours ago, Benson said: Dec 30 20:01:55 UnRAID-Backup kernel: pci 0000:08:00.0: 16.000 Gb/s available PCIe bandwidth, limited by 2.5 GT/s x8 link at 0000:00:01.0 (capable of 63.008 Gb/s with 8 GT/s x8 link) Quote Link to comment
jbartlett Posted January 9, 2020 Share Posted January 9, 2020 @icemansid - Please upload/email a debug file from the DiskSpeed app (link at the bottom of the page). Use the left button to create a regular/smaller report. The link speed uses the results from a lspci -vmm command and the debug file will have what it returned. Quote Link to comment
icemansid Posted January 9, 2020 Author Share Posted January 9, 2020 (edited) 44 minutes ago, jbartlett said: @icemansid - Please upload/email a debug file from the DiskSpeed app (link at the bottom of the page). Use the left button to create a regular/smaller report. The link speed uses the results from a lspci -vmm command and the debug file will have what it returned. File as requested. file_20200109_123133.tar.gz Edited January 9, 2020 by icemansid err small file. Quote Link to comment
jbartlett Posted January 9, 2020 Share Posted January 9, 2020 (edited) On 1/8/2020 at 12:33 PM, Benson said: Dec 30 20:01:55 UnRAID-Backup kernel: pci 0000:08:00.0: 16.000 Gb/s available PCIe bandwidth, limited by 2.5 GT/s x8 link at 0000:00:01.0 (capable of 63.008 Gb/s with 8 GT/s x8 link) Here's the controller link report for the 0000:08:00.0 device 08:00.0 Serial Attached SCSI controller: Broadcom / LSI SAS2308 PCI-Express Fusion-MPT SAS-2 (rev 05) Capabilities: [68] Express (v2) Endpoint, MSI 00 LnkCap: Port #0, Speed 8GT/s, Width x8, ASPM L0s, Exit Latency L0s <64ns, L1 <1us ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+ LnkSta: Speed 8GT/s, Width x8, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt- I have a lookup table that for PCI3 with a transfer rate of 8 and a width of 8 returns "7.88 GB/s". It looks like this reference table is in error. Based on information from https://paolozaino.wordpress.com/2013/05/21/converting-gts-to-gbps/ I should be able to compute it. The transfer rate of 8 GT/s identifies it as a PCI 3 controller. 8 GT/s multiplied by 8 lanes equals 64 GT/s, multiplied by the overhead (LineCodeL/LineCodeH) of 128b/130b equals 63.02 GB/s. Edited January 9, 2020 by jbartlett Quote Link to comment
JorgeB Posted January 10, 2020 Share Posted January 10, 2020 3 hours ago, jbartlett said: Here's the controller link report for the 0000:08:00.0 device OK, so looks like diskspeed is reporting the correct speed, or at least the same as lspci, the problem might be the server bios. Quote Link to comment
icemansid Posted June 2, 2020 Author Share Posted June 2, 2020 I just wanted to provide an update to this posting - as it turns out, the issue was with SMR drives being used in the system. While we removed the 9206 cards, it wasn't until we moved the SMR drives out of the parity that we were able to clear the issue. Quote Link to comment
peterg23 Posted June 2, 2020 Share Posted June 2, 2020 I posted some information on the current SMR situation a few days ago here: Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.