Reconstruct write slower than read/modify/write

[Vetl]

Hello everyone,

 

I'm hoping someone can help out to troubleshoot where the bottleneck is on my system. As expected read/write/modify was slow since the beginning, so I switched to reconstruct method, it was working pretty well for some time, I missed the point where it degraded ( adding more HDDs into array). So now my speeds are around 40mb/s, with speed being faster in read/write/modify mode and slower in reconstruct. I did speedtest of drives, my parity drive is fast, I know my 4tbs are not great but, when connected to windows machine I can read/write 120mb/s, so much faster than 40mb/s. 

matrix-diagnostics-20240117-2155_reconst.zip matrix-diagnostics-20240117-2152_rd,wr,md.zip

[JorgeB]

Post the output of:

lspci -d 1000: -vv

 

[itimpi]

The only reasons I can think of are:

• The system is having problems reading at least one of the drives do resets are slowing things down.
• the Disk controller is limiting throughput when all drives are active.

 

[Vetl]

2 hours ago, JorgeB said:

Post the output of:

lspci -d 1000: -vv

 

please see the attached file ( was captured during read/modify/write mode if it makes difference)

lspci.txt

[Vetl]

1 hour ago, itimpi said:

The only reasons I can think of are:

• The system is having problems reading at least one of the drives do resets are slowing things down.
• the Disk controller is limiting throughput when all drives are active.

 

well, the speed tests show pretty much similar reads on all drives, for the second part I don't know how to validate

[JorgeB]

LnkSta: Speed 5GT/s (downgraded), Width x1 (downgraded)

 

Controller is linking at PCIe 2.0 instead of 3.0, and worse than that at x1, instead of x8, so bandwidth is being severely limited, see if you have a better slot where you could use the HBA.

[Vetl]

2 hours ago, JorgeB said:

LnkSta: Speed 5GT/s (downgraded), Width x1 (downgraded)

 

Controller is linking at PCIe 2.0 instead of 3.0, and worse than that at x1, instead of x8, so bandwidth is being severely limited, see if you have a better slot where you could use the HBA.

Someone plugged the card into the only one PCIe 2.0 slot on the MB lol moved to the PCIe 3.0 x8 slot, and now getting the following:
LnkSta: Speed 8GT/s, Width x2 (downgraded)

 

seems like "Width x2 (downgraded)" part doesn't look good, should I be concerned ? if yes what would be the next steps?

seems like reconstruct writing is back to 120-130mb/s which max fox those hhds. How many HHDs can that controller handle ?

 

lspci_3.txt

Edited January 18 by Vetl

[JorgeB]

41 minutes ago, Vetl said:

seems like "Width x2 (downgraded)" part doesn't look good

Not great, but it still has x4 times the bandwidth as before, and good for roughly 1700/1800MB/s, divide that for the number of disks and you can see when it can become a bottleneck.

[Vetl]

17 minutes ago, JorgeB said:

Not great, but it still has x4 times the bandwidth as before, and good for roughly 1700/1800MB/s, divide that for the number of disks and you can see when it can become a bottleneck.

pretty soon at 10 disks in the array pushing about 120mb/s and I'm only about 50% in my data migration, this is 46 bay case from microfocus. I have tried to put CPU in performance mode, II0 PCIe settings from auto to x8 on those ports but still getting "Width x2 (downgraded)" any way to push it for x8 ?

[JorgeB]

It should be automatic as long as it's really an x8 slot, you can't force it, note that it can be x8 physically but only x8 electrically, check the board manual. or try a different one.

[Vetl]

1 hour ago, JorgeB said:

It should be automatic as long as it's really an x8 slot, you can't force it, note that it can be x8 physically but only x8 electrically, check the board manual. or try a different one.

hmmm tried all x8 pcie 3.0 ports with the same result, if used x16 it downgraded to Width x1. Could it be that I'm not using bandwidth and it scaled down ? i.e. when I will be hitting limit it will scale up? otherwise I guess I will have to add another card when I will start hitting the limit. Will adding another card impact the array ?

[JorgeB]

18 minutes ago, Vetl said:

Could it be that I'm not using bandwidth and it scaled down ? i.e. when I will be hitting limit it will scale up?

No, it links initially and doesn't change, looking at the board specs:

2 PCI-E 3.0 x16,
3 PCI-E 3.0 x8
1 PCI-E 2.0 x4 (in x8 slot)

 

All three PCIe 3.0 slots should link with the HBA @ x8, the PCIe 2.0 slot should link @ x4, so possibly a controller issue, I think it would be unlikely that the board/CPUs would be the problem, but not impossible.