1 hour ago, icemansid said:

old drives connected to this controller will cause drive speed issues across all of the drives.

I don't think so, but you can troubleshoot in this direction.

 

As mention before, HBA should have PCIe switch and your mainboard also have PCIe switch, those were in cascade and may have compatible problem.

 

From Syslog, this have a sign, LSI HBA only running in PCIe Gen 1 speed, but bandwidth still have 16Gb/s. ( Even all problem solve, throughput will ceiling at this too )

Dec 30 20:01:55 UnRAID-Backup kernel: pci 0000:08:00.0: 16.000 Gb/s available PCIe bandwidth, limited by 2.5 GT/s x8 link at 0000:00:01.0 (capable of 63.008 Gb/s with 8 GT/s x8 link)

 

I also notice you have P420i HBA, 1 or 2 disk connect, could you unplug it and try.

 

Which Expander you got ??
 

 

Edited January 8, 20206 yr by Benson

I am using the HP 468405-002 and using a PCI-e to USB3 adpater to power it.  The p420i is the onboard running the HP OEM dirves in a raid array acting as a cache drive.  Ive rebooted and installed a second SAS2308 card for testing as well.  Ive also attached a fresh diagnostic report.

 

 

I dont exactly follow what you mean by PCIe switch.

unraid-backup-diagnostics-20200108-1436.zip

From new Syslog, the second SAS2308 card also run at PCIe Gen 1 speed. In fact all device were in Gen 1, but machine support PCIe Gen 3. Does you wrongly set the limit in BIOS ?

 

25 minutes ago, icemansid said:

I dont exactly follow what you mean by PCIe switch.

It like expander to provide a function to let you connect more device but share same bandwidth, just expander service for SAS device.
 

 

How about the result on 2nd 2308 HBA ??

Edited January 8, 20206 yr by Benson

3 hours ago, Benson said:

From new Syslog, the second SAS2308 card also run at PCIe Gen 1 speed. In fact all device were in Gen 1, but machine support PCIe Gen 3. Does you wrongly set the limit in BIOS ?

 

It like expander to provide a function to let you connect more device but share same bandwidth, just expander service for SAS device.
 

 

How about the result on 2nd 2308 HBA ??

2nd 2308 gives very similar results at this point.  Nothing in the BIOS stands out but I have an HPE storage engineer looking at it as well. Hopefully he can assist with the PCI negotiations..

11 hours ago, Benson said:

Dec 30 20:01:55 UnRAID-Backup kernel: pci 0000:08:00.0: 16.000 Gb/s available PCIe bandwidth, limited by 2.5 GT/s x8 link at 0000:00:01.0 (capable of 63.008 Gb/s with 8 GT/s x8 link)

I didn't even look at the diags for this since diskspeed docker should report the correct link speed and it was reporting PCIe 3.0, @jbartlettmight want to take a look at this.

The DiskSpeed app simply reports what the dd command is outputting. If you're having issues reading all the drives at once, try removing half of the drives and then run a controller benchmark to see if you get the same results or better. If the same, remove the existing and hook the others back up. If better, then add a couple drives and do another benchmark. Rinse & repeat. See if there's a magic number or a given drive that's causing the issue.

1 hour ago, jbartlett said:

The DiskSpeed app simply reports what the dd command is outputting.

I meant the link speed:

 

 

13 hours ago, Benson said:

Dec 30 20:01:55 UnRAID-Backup kernel: pci 0000:08:00.0: 16.000 Gb/s available PCIe bandwidth, limited by 2.5 GT/s x8 link at 0000:00:01.0 (capable of 63.008 Gb/s with 8 GT/s x8 link)

 

@icemansid - Please upload/email a debug file from the DiskSpeed app (link at the bottom of the page). Use the left button to create a regular/smaller report.

 

The link speed uses the results from a lspci -vmm command and the debug file will have what it returned.

44 minutes ago, jbartlett said:

@icemansid - Please upload/email a debug file from the DiskSpeed app (link at the bottom of the page). Use the left button to create a regular/smaller report.

 

The link speed uses the results from a lspci -vmm command and the debug file will have what it returned.

File as requested.

 

file_20200109_123133.tar.gz

Edited January 9, 20206 yr by icemansid
err small file.

On 1/8/2020 at 12:33 PM, Benson said:

Dec 30 20:01:55 UnRAID-Backup kernel: pci 0000:08:00.0: 16.000 Gb/s available PCIe bandwidth, limited by 2.5 GT/s x8 link at 0000:00:01.0 (capable of 63.008 Gb/s with 8 GT/s x8 link)

Here's the controller link report for the 0000:08:00.0 device

08:00.0 Serial Attached SCSI controller: Broadcom / LSI SAS2308 PCI-Express Fusion-MPT SAS-2 (rev 05)
	Capabilities: [68] Express (v2) Endpoint, MSI 00
		LnkCap:	Port #0, Speed 8GT/s, Width x8, ASPM L0s, Exit Latency L0s <64ns, L1 <1us
			ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
		LnkSta:	Speed 8GT/s, Width x8, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-

I have a lookup table that for PCI3 with a transfer rate of 8 and a width of 8 returns "7.88 GB/s". It looks like this reference table is in error.

 

Based on information from https://paolozaino.wordpress.com/2013/05/21/converting-gts-to-gbps/ I should be able to compute it. The transfer rate of 8 GT/s identifies it as a PCI 3 controller. 8 GT/s multiplied by 8 lanes equals 64 GT/s, multiplied by the overhead (LineCodeL/LineCodeH) of 128b/130b equals 63.02 GB/s.

 

Edited January 9, 20206 yr by jbartlett

3 hours ago, jbartlett said:

Here's the controller link report for the 0000:08:00.0 device

OK, so looks like diskspeed is reporting the correct speed, or at least the same as lspci, the problem might be the server bios.

I just wanted to provide an update to this posting - as it turns out, the issue was with SMR drives being used in the system. While we removed the 9206 cards, it wasn't until we moved the SMR drives out of the parity that we were able to clear the issue.  

I posted some information on the current SMR situation a few days ago here: