Hey peeps, I have some questions about what I can do to improve parity check speeds with my server.

 

I'm currently running a Supermicro X9DRD-7LN4F mobo with dual xeon E5-2697 v2 and 128g ram. 2 supermicro 846 chassis, one with 24 bays and the mobo and another with 36 bays just as a jbod. Both have sas2 expanders in them and are full with 60 drives. 2 HBA sas2 2008 controllers, one internal ports and one with external ports (connected with dac cables). 30 x 4tb sata drives in the array and the rest in zfs pools. Obviously I'm limited with the sata and sas2 expanders but I'm starting to expand space with 12tb sas drives hence now seeing speed issues. Parity checks take 24hrs+ at an average of 50MB/s but never concerned me before with them being 4tb drives. Now that I'm adding in 12tb drives it's becoming an issue.

 

Question 1: when upgrading the parity drives to 12tb I'm seeing 80MB/s during the reads of 28 4tb drives (bandwidth being the issue here see next question) but when the 4tb drives have finished and it's just doing the 8tb parity drive expansion part then I am only getting a max of 75MB/s speeds on the 2 drives, where is the bottleneck here?

 

Question 2: with all these drives I believe I'm hitting a bandwidth issue with all array drives running and hitting a max of 2500MB/s total. What would be the best upgrade path to increase the total bandwidth available and speed up parity operations?

 

I've attached diags to help.server-diagnostics-20250329-1056.zip

 

Many thanks for your thoughts  

SAS2 expander with dual ink will be limited to 2400MB/s max, you could possibly get better performance during parity check/rebuild if you split the array drives between both chassis, half in each, that should give you 160MB/s max, but see below.

 

1 hour ago, TheThingIs said:

then I am only getting a max of 75MB/s speeds on the 2 drives, where is the bottleneck here?

Parity check is single thread, and with so many devices it can also be CPU/memory bandwidth limited, and with parity2 installed, that limit applies even when some/all the other disks are not being read, since they still must be included in the calculation for the whole parity, as all zeroes, of course, and with the CPU you have, 80MB/s could be around the max speed with 30 devices, I have a test server, with a Xeon E5-V3 and DDR4, so a little faster than yours, and I'm limited to 160MB/s with 30 devices, and it's not a device/controller bottleneck, since that limit for that server is about twice that.

splitting the drives is a good idea as they are currently, apart from 2, all on one chassis.

 

I'm open to upgrading everything, mobo, backplanes, hba cards...although it sounds like you think the mobo would be my starting point?

I'm not rich, so I'd start with replacing the expanders with SAS3 ones, like the AEC 82885 T which are really cheap, to double the bandwidth between them, and the controller with a newer one as well, mostly because of PCIe2 vs PCIe3 of the board, probably a 9400-16i since the 9300 uses too much power, the 9305 is too expensive, and less controllers means less to replace if something breaks.

 

During all this I'd also start replacing 4TB drives with 12TB ones, moving the contents of 2-3 4TB drives (2 if more space needed to expand, 3 if not) to keep the array in one case.

 

You also didn't mention using the 2308 controller on the board, I'd really start with lashing that to IT mode and start using that, and removing one of the 2008, because again PCIe2 vs PCIe3.