Hollander_1908 Posted December 12, 2023 Share Posted December 12, 2023 Hello everyone, I've been experiencing consistently low parity check speeds for a while now, and although it's not extremely slow (around 50MB/s), I've recently noticed that some of my friends are achieving speeds around 200MB/s. Naturally, I'd like to optimize my setup to match these higher speeds. I'm unsure about the factors causing the slower speeds on my end, which is why I've created this post. I've attached the diagnostics for your reference. If you need any additional information, please don't hesitate to ask! Thanks in advance! diagnostics-20231212-1545.zip Quote Link to comment
Zonediver Posted December 12, 2023 Share Posted December 12, 2023 (edited) 12 minutes ago, Hollander_1908 said: Hello everyone, I've been experiencing consistently low parity check speeds for a while now, and although it's not extremely slow (around 50MB/s), I've recently noticed that some of my friends are achieving speeds around 200MB/s. Naturally, I'd like to optimize my setup to match these higher speeds. I'm unsure about the factors causing the slower speeds on my end, which is why I've created this post. I've attached the diagnostics for your reference. If you need any additional information, please don't hesitate to ask! Thanks in advance! diagnostics-20231212-1545.zip 184.62 kB · 0 downloads 50MB/sec is a little bit low, but it depends what the server is doing in the background during the parity check. My Sys reaches normaly ~170MB/s but if someone is using Plex, the rate drops to ~145MB/s. Maybe a bottleneck on the controller(s)? There is a Docker application called "DiskSpeed" - install it and test your throughput of HDDs and Controllers 👍 Edited December 12, 2023 by Zonediver Quote Link to comment
Solution JorgeB Posted December 12, 2023 Solution Share Posted December 12, 2023 You are using a controller with a SATA port multiplier, and there are 5 disks on the same port, so bandwidth of a single SATA port is shared by all 5. Quote Link to comment
Hollander_1908 Posted December 12, 2023 Author Share Posted December 12, 2023 47 minutes ago, JorgeB said: You are using a controller with a SATA port multiplier, and there are 5 disks on the same port, so bandwidth of a single SATA port is shared by all 5. I honestly didn't know it had a multiplier. I have 10x 12TB HDD and 2x 1TB SSD. Now that you mention it, I'll be optimizing my setup by connecting 6 HDDs to the motherboard, using 4 HDDs on the RAID controller, and attaching 2 SSDs to the multiplier. The SSDs on the multiplier are chosen because of their higher write speeds, and both are serving as a cache in a single pool anyway. I will make the changes tonight, run some tests, and provide feedback on the impact on speed and overall performance. Thanks! Quote Link to comment
Hollander_1908 Posted December 12, 2023 Author Share Posted December 12, 2023 1 hour ago, Zonediver said: 50MB/sec is a little bit low, but it depends what the server is doing in the background during the parity check. My Sys reaches normaly ~170MB/s but if someone is using Plex, the rate drops to ~145MB/s. Maybe a bottleneck on the controller(s)? There is a Docker application called "DiskSpeed" - install it and test your throughput of HDDs and Controllers 👍 I installed the 'DiskSpeed' Docker and conducted some tests. Interestingly, individual speeds reached around 200MB/s. It appears that the issue highlighted by @JorgeB might indeed be the key to optimizing the performance. Thanks for suggesting the docker – it seems to be a valuable tool. See screenshot of the tests on all my HDD drives for more details. Quote Link to comment
ezhik Posted December 12, 2023 Share Posted December 12, 2023 (edited) Make sure this is set in Settings ---> Disk Settings Tunable (md_write_method): reconstruct write Edited December 12, 2023 by ezhik Quote Link to comment
Kilrah Posted December 12, 2023 Share Posted December 12, 2023 1 minute ago, ezhik said: Make sure this is set in "Settings" -> "Disk settings" Tunable (md_write_method): reconstruct write It does not affect parity operations. Quote Link to comment
ezhik Posted December 12, 2023 Share Posted December 12, 2023 (edited) 3 minutes ago, Kilrah said: It does not affect parity operations. Ah, right. Then I would check the "write-cache" settings on the HDDs ### for SATA hdparm -W /dev/sd[a-z] # Sample output: /dev/sdb: write-caching = 1 (on) ### for SATA+SAS sdparm --get=WCE /dev/sd[a-z] # Sample output: WCE 1 [cha: y, def: 1] /dev/sdc: ATA ADATA SU800 8B NOTE: WCE = 1 means write cache is enabled You can also SET the "write-cache" bit to enabled (and save): # Set Write-Cache and SAVE the setting to persist upon reboot sdparm --set=WCE --save $disk Edited December 12, 2023 by ezhik Quote Link to comment
Hollander_1908 Posted December 12, 2023 Author Share Posted December 12, 2023 1 minute ago, ezhik said: Make sure this is set in "Settings" -> "Disk settings" Tunable (md_write_method): reconstruct write I had already taken note of this setting and ensured that it was configured correctly. I can confirm that it is indeed set up as needed. Thanks anyway, appreciate it! Quote Link to comment
Hollander_1908 Posted December 12, 2023 Author Share Posted December 12, 2023 1 minute ago, ezhik said: Ah, right. Then I would check the "write-cache" settings on the HDDs ### for SATA # hdparm -W /dev/sd[a-z] Sample output: /dev/sdb: write-caching = 1 (on) ### for SATA+SAS sdparm --get=WCE /dev/sd[a-z] Sample output: WCE 1 [cha: y, def: 1] /dev/sdc: ATA ADATA SU800 8B WCE = 1 means write cache is enabled Not entirely certain, but could this potentially impact the speed of the parity check too? Otherwise, I might start by changing the ports first. Quote Link to comment
ezhik Posted December 12, 2023 Share Posted December 12, 2023 1 minute ago, Hollander_1908 said: Not entirely certain, but could this potentially impact the speed of the parity check too? Otherwise, I might start by changing the ports first. Absolutely. Make sure your write-cache is enabled, this helps with sustained writes. Quote Link to comment
JorgeB Posted December 12, 2023 Share Posted December 12, 2023 Diskspeed test also has a controller benchmark option, that will show the bottleneck. Quote Link to comment
pixels703 Posted December 12, 2023 Share Posted December 12, 2023 I suspect this is related to PCIe lanes. Check the number of PCIe lanes with where each of your cards are plugged. If you’re running a card on an X1 lane even though it’s an X16 slot, you’re limiting throughput. And ensure that you have enough lanes on the CPU to accommodate the lanes being used. Go to the mobo page and look at PCI_E1, *_E2, etc etc. it will show slot size and allocated lanes. E.g., PCI_E1 x16 = 16 lanes. x4 = 4 lanes. Many mobos have x16 slots that only run x1 lane. Many mobos run the m.2 through a chip and you have limitations there. Then look at version of PCIe - 2.0 runs 500MB/s and 3.0 runs 1GB/s. So a x4 with a v2 card only runs 2GB/s. Spread that over the number of drives and that’s why it’s slow on parity. Just a suggestion. Sent from my iPhone using Tapatalk 2 Quote Link to comment
Hollander_1908 Posted December 15, 2023 Author Share Posted December 15, 2023 Hey guys, Upon investigating further on the same evening I started this topic, I reconfigured the disk connections and discovered that two out of the six SATA ports on my motherboard were completely non-functional. It appears that this was probably the reason why I initially arranged the disk connections in a certain way. Unfortunately, due to the malfunctioning of these two ports, I was limited in terms of reconfiguring the setup. To address this issue, I proceeded to order an M.2 to SATA adapter equipped with an ASM1166 chip, providing functionality for six SATA ports. I received this tiny piece of equipment yesterday and promptly connected the drives to it. I am pleased to report that I am now achieving the desired speed of 200MB/s, effectively resolving the problem I was experiencing. I would like to express my gratitude for all the suggestions provided by the community. Your insights were instrumental in identifying and troubleshooting the problem. Cheers! 3 Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.