I am in the process of upgrading my parity drives to 2x 24TB drives. I am currently running a parity sync on both drives at the same time. I am noticing that my speeds are lackluster compared to the speeds I get when benchmarking with the diskspeed app.

 

My parity sync speeds are consistently between 90-100MB/s. However, my individual benchmarked speeds are fairly close to what I would expect as shown in the graph below. Also my controller benchmarks are generally 5-10% lower than the sum of the individual speed benchmarks, which is still much faster than the parity sync speeds I am getting.

 

(Graph taken with 5% increment benchmarks, diskspeed default is 10%)

 

I am aware that parity syncs and checks are bottlenecked by the slowest drive, but even if that's the case I would only expect the 90-100 MB/s speeds toward the end of the 8, 10, 12, and 14TB disks.

 

So my question is, why would my parity syncs hit a ceiling early on? I expected to see a gradual slowdown as the 8TB disks were processed, but instead it stayed steady at ~2.5 GB/s nearly the entire time.

 

(Parity sync, 8 TB disks ended near the end of graph, at the small bump. Dip earlier on was due to a short stint of disk reads)

Any ideas what could be bottlenecking and how I might diagnose?

tower-diagnostics-20240905-0039.zip

Edited September 5, 20241 yr by johnsanc

Run the diskspeed controller tests and post those.

Here are the benchmarks i took immediately before starting parity sync. No drives were moved from these locations on the controllers.

Those show a slight controller bottleneck for some disks, but not enough to justify the speed, the more drives you have, the more it will tax the CPU, parity2 also adds to that, and parity check/sync is single threaded, with 28 drives and dual parity I got around 125MB/s max with my server, never tested with Ryzen, so possibly you are just being limited by the CPU.

That’s what I suspected as well, but I don’t see any crazy cpu utilization on any specific thread so I wasn’t sure.

 

Once my parity sync is complete in a couple days I’ll run a few more tests

 

 

Quick update:

0-8 TB: ~90-95 MB/s

8-10 TB: ~95-105 MB/s

10-12 TB: ~105-110 MB/s

12-14 TB: ~110-115 MB/s

 

I am currently in the 12-14 TB range with 10 data disks in scope. Overall as you can see speed is increasing to a more respectable rate given the disk position as certain disk size thresholds are passed.

Yeah, but with dual parity there's still a performance penalty for the other disks, since they are still part of the calculations, there wouldn't be with single parity, i.e., you should notice much larger performance gains.

I think the issue may be CPU related due to single-threaded nature of parity checks. After my sync was complete I added 2 more data disks and the check was only going at about 90-95 MB/s. I noticed that unraidd0 process was using 95% cpu according to top. I rebooted, tried a check again and now its going at 105-110 MB/s steady which is an improvement.

 

I suppose its not a high priority, but has there been any consideration to multithread the parity sync/check?

I'm not sure, but not aware of that being on the radar for the near future.