Hello everyone,

 

I've been experiencing consistently low parity check speeds for a while now, and although it's not extremely slow (around 50MB/s),

I've recently noticed that some of my friends are achieving speeds around 200MB/s.

 

Naturally, I'd like to optimize my setup to match these higher speeds.

I'm unsure about the factors causing the slower speeds on my end, which is why I've created this post.

 

I've attached the diagnostics for your reference.

If you need any additional information, please don't hesitate to ask!

 

Thanks in advance!

diagnostics-20231212-1545.zip

12 minutes ago, Hollander_1908 said:

Hello everyone,

 

I've been experiencing consistently low parity check speeds for a while now, and although it's not extremely slow (around 50MB/s),

I've recently noticed that some of my friends are achieving speeds around 200MB/s.

 

Naturally, I'd like to optimize my setup to match these higher speeds.

I'm unsure about the factors causing the slower speeds on my end, which is why I've created this post.

 

I've attached the diagnostics for your reference.

If you need any additional information, please don't hesitate to ask!

 

Thanks in advance!

diagnostics-20231212-1545.zip 184.62 kB · 0 downloads

 

50MB/sec is a little bit low, but it depends what the server is doing in the background during the parity check.

My Sys reaches normaly ~170MB/s but if someone is using Plex, the rate drops to ~145MB/s.

Maybe a bottleneck on the controller(s)?

There is a Docker application called "DiskSpeed" - install it and test your throughput of HDDs and Controllers 👍

Edited December 12, 20232 yr by Zonediver

You are using a controller with a SATA port multiplier, and there are 5 disks on the same port, so bandwidth of a single SATA port is shared by all 5.

47 minutes ago, JorgeB said:

You are using a controller with a SATA port multiplier, and there are 5 disks on the same port, so bandwidth of a single SATA port is shared by all 5.

 

I honestly didn't know it had a multiplier. I have 10x 12TB HDD and 2x 1TB SSD. Now that you mention it, I'll be optimizing my setup by connecting 6 HDDs to the motherboard, using 4 HDDs on the RAID controller, and attaching 2 SSDs to the multiplier. The SSDs on the multiplier are chosen because of their higher write speeds, and both are serving as a cache in a single pool anyway. I will make the changes tonight, run some tests, and provide feedback on the impact on speed and overall performance. Thanks!

1 hour ago, Zonediver said:

 

50MB/sec is a little bit low, but it depends what the server is doing in the background during the parity check.

My Sys reaches normaly ~170MB/s but if someone is using Plex, the rate drops to ~145MB/s.

Maybe a bottleneck on the controller(s)?

There is a Docker application called "DiskSpeed" - install it and test your throughput of HDDs and Controllers 👍

I installed the 'DiskSpeed' Docker and conducted some tests. Interestingly, individual speeds reached around 200MB/s. It appears that the issue highlighted by @JorgeB might indeed be the key to optimizing the performance. Thanks for suggesting the docker – it seems to be a valuable tool.

See screenshot of the tests on all my HDD drives for more details.

Make sure this is set in

Settings ---> Disk Settings

 

Tunable (md_write_method): reconstruct write

 

Edited December 12, 20232 yr by ezhik

1 minute ago, ezhik said:

Make sure this is set in "Settings" -> "Disk settings"

 

Tunable (md_write_method): reconstruct write

 

It does not affect parity operations.

3 minutes ago, Kilrah said:

It does not affect parity operations.

 

Ah, right.

 

Then I would check the "write-cache" settings on the HDDs

 

### for SATA

hdparm -W /dev/sd[a-z]

# Sample output:

/dev/sdb:
 write-caching =  1 (on)

### for SATA+SAS
sdparm --get=WCE /dev/sd[a-z]

# Sample output:
WCE           1  [cha: y, def:  1]
    /dev/sdc: ATA       ADATA SU800       8B  

 

NOTE: WCE = 1 means write cache is enabled

 

You can also SET the "write-cache" bit to enabled (and save):

 

# Set Write-Cache and SAVE the setting to persist upon reboot

sdparm --set=WCE --save $disk

 

Edited December 12, 20232 yr by ezhik

1 minute ago, ezhik said:

Make sure this is set in "Settings" -> "Disk settings"

 

Tunable (md_write_method): reconstruct write

 

I had already taken note of this setting and ensured that it was configured correctly. I can confirm that it is indeed set up as needed. Thanks anyway, appreciate it!

1 minute ago, ezhik said:

 

Ah, right.

 

Then I would check the "write-cache" settings on the HDDs

 

### for SATA

# hdparm -W /dev/sd[a-z]

Sample output:

/dev/sdb:
 write-caching =  1 (on)

### for SATA+SAS
sdparm --get=WCE /dev/sd[a-z]

Sample output:
WCE           1  [cha: y, def:  1]
    /dev/sdc: ATA       ADATA SU800       8B  

WCE = 1 means write cache is enabled

 

Not entirely certain, but could this potentially impact the speed of the parity check too? Otherwise, I might start by changing the ports first.

1 minute ago, Hollander_1908 said:

Not entirely certain, but could this potentially impact the speed of the parity check too? Otherwise, I might start by changing the ports first.

 

Absolutely. Make sure your write-cache is enabled, this helps with sustained writes. 

Diskspeed test also has a controller benchmark option, that will show the bottleneck.

I suspect this is related to PCIe lanes.

Check the number of PCIe lanes with where each of your cards are plugged. If you’re running a card on an X1 lane even though it’s an X16 slot, you’re limiting throughput.

And ensure that you have enough lanes on the CPU to accommodate the lanes being used.

Go to the mobo page and look at PCI_E1, *_E2, etc etc. it will show slot size and allocated lanes. E.g., PCI_E1 x16 = 16 lanes. x4 = 4 lanes. Many mobos have x16 slots that only run x1 lane. Many mobos run the m.2 through a chip and you have limitations there.

Then look at version of PCIe - 2.0 runs 500MB/s and 3.0 runs 1GB/s. So a x4 with a v2 card only runs 2GB/s. Spread that over the number of drives and that’s why it’s slow on parity.

Just a suggestion.


Sent from my iPhone using Tapatalk

Hey guys,

Upon investigating further on the same evening I started this topic, I reconfigured the disk connections and discovered that two out of the six SATA ports on my motherboard were completely non-functional. It appears that this was probably the reason why I initially arranged the disk connections in a certain way. Unfortunately, due to the malfunctioning of these two ports, I was limited in terms of reconfiguring the setup.

 

To address this issue, I proceeded to order an M.2 to SATA adapter equipped with an ASM1166 chip, providing functionality for six SATA ports.

 

 

I received this tiny piece of equipment yesterday and promptly connected the drives to it.

I am pleased to report that I am now achieving the desired speed of 200MB/s, effectively resolving the problem I was experiencing.

 

I would like to express my gratitude for all the suggestions provided by the community.

Your insights were instrumental in identifying and troubleshooting the problem.

 

Cheers!