Is parity-check/-rebuild single threaded?

[hawihoney]

From my own experience it looks as if parity-check/-rebuild runs single threaded? Is this true?

 

[JorgeB]

1 hour ago, hawihoney said:

if parity-check/-rebuild runs single threaded? Is this true?

Yes.

[hawihoney]

Thanks.

 

Seems that single-thread performance is more important then on Unraid parity-check/-rebuild than number of CPUs, cores.

 

[Pixel5]

How many disks are in your array that the CPU is pegged at 100% in this scenario?

 

im running a fairly small array with 4 disks and also have a cpu with high single clock performance so i never even noticed that it puts any load on the CPU, at which number of disks could this become the limiting factors instead of the drive speed itself?

[itimpi]

On most cases parity-check is input/output limited by the speeds of the disks rather than limitations of the CPU.

[jpporter]

I just spent the last few months updating my Unraid system with an updated Supermicro case with 12GB SAS expanders, 12 more drive bays, bought new 2x 12gb HBAs, updated the CPU to lower power, lower speed (but with 4 more cores)...  Did all the testing, using old crappy drives, assuming they were the reason parity was slow.  Went ahead and moved over the real drives... the end result was 1/3 the speed of parity check, from ~24 hours to 3+ days...  Today I figured out that the parity check is single threaded.  Just bought some overkill CPUs that are twice the clockrate and 70% more power usage.  I'll try to reply once I get the CPUs replaced.

 

In short, it would be nice if the parity check/build was multithreaded.

[John_M]

On 2/22/2021 at 10:45 PM, jpporter said:

In short, it would be nice if the parity check/build was multithreaded.

 

Why? How would that speed things up?

[trurl]

Parity is completely I/O bound so CPU cycles aren't an issue except for very old CPUs on dual parity.

 

If you installed port multipliers that probably explains your slowdown.

[Jan Hoker]

On 3/3/2021 at 10:27 PM, trurl said:

Parity is completely I/O bound so CPU cycles aren't an issue except for very old CPUs on dual parity.

 

If you installed port multipliers that probably explains your slowdown.

I cannot agree with you. I have similar experience as jpporter. I have Ryzen 3700X thats not slow CPU, 2x1TB Samsung 980 PRO NVMe as chace drives and 3x12TB WD Gold HDDs one of them parity drive. Every time when parity check starts one thread is pegged to 100% and all my work must stop, shares are usually not accesible during that time and services like Plex or HOOBS are not working or at very limited speed/laggy. That includes shares which are only in cache drives(game libraries), because CPU is struggling with that kind of load. It would be awesome if this proces would be multitreaded, not because speeding up, i dont care how long it takes, but because I could access data. It's maybe issue just for Ryzen CPUs caused by architecture and one 100% pegged thread can cause this trouble. But I'm really unsatisfied by performance during parity check.

[primeval_god]

I would suggest taking a closer look at you system performance with something like Netdata, the unRAID dashboard is not particularly detailed about system performance. I would suspect you will find very high iowait and that your pegged core is mostly busy waiting for IO. Why that would bog down your entire system and what to do about it I unfortunately dont known.

[utilitybee]

Just found this thread because i am running into the same issue. I was wondering why i was only getting 89MBs and the type 260MBs of my drives (mg09aca18te) i do have 30 of them in a pool with 2 parity. One core of my Gold 6230R CPU is pegged at 100% for 3 to 4 days to do a parity check. I have almost no iowait on avg. Since it is limited to a single core i only get a total read speed of the array of 2.6 GB/s when it should be close to 7.6 GB/s. All hdds run though 3x LSI 9305 and have dedicated pci lanes so the throughput should not be a issue. Am i missing something or this just something i am going to have to live with?

 

Edit: so after trying to get as much single thread performance as possible I found out that turbo is not enabled by default on my motherboard. Now with all that squared away i got up 110MBs per drive and 3.2 GB/s array.

Edited July 5 by utilitybee

[JorgeB]

8 hours ago, utilitybee said:

I was wondering why i was only getting 89MBs

That still seems low, last time I testes I could get more than that with an inferior CPU (single thread rating), please post the diagnostics and the output of

lspci -d 1000: -vv

to see if anything jumps out.

[utilitybee]

Thank you for coming back to a 2 year other thread! I provided the files you asked for.  I have been running unraid since 2015 this is the first time i have been stumped. 😞

 

tower-diagnostics-20240706-0508.zip lspci -d 1000 -vv output.txt

[JorgeB]

Links speed/width is OK for all HBAs, let me re-test the parity check speed in my test server with v6.12.10, since it's been a while since I tested, and not with the latest releases, I'll get back to you.

[JorgeB]

Did a quick test and speeds with v6.12.10 look normal to me, this was with a 30 device array, most devices are connected to LSI 9308 HBAs, one of the HBAs is connected to a 12 port expander, a couple of the devices are connected to the onboard SATA, I see an average of 5GB/s, with some peaks of 5.2GB/s (170 to 180MB/s per drive):

 

 

CPU is a Xeon E5-1620 v3, which compared to yours, and according to passmark, has a a lower single thread score of about 10%, so something else may be going on, recommend running the diskspeed container controller test for all controllers in case there's an issue with one of them, despite normal link speeds, this test:

 

[utilitybee]

Thank you for the update! I did benchmark everything like you asked. I don't see anything out of the ordinary. All ssds are on their own controller.

Each controller need 4.7GBs so brings the total to 9.4GBs for a parity check?

Edited July 7 by utilitybee

[JorgeB]

Everything looks good there, you are certain that there wasn't anything else accessing the array during the check right?

[utilitybee]

I was pretty sure there was nothing but i wanted to make sure. So i put it in Maintenance Mode and got the exact same result. 😢

[JorgeB]

I don't see anything wrong, do a quick CPU test to see if the CPU is performing normally:

 

time $(i=0; while (( i < 9999999 )); do (( i ++ )); done)

 

Post the output of that, should take less than a minute.

[utilitybee]

real    0m17.709s
user    0m17.709s
sys     0m0.000s

 

this is with a running server. like vms and dockers.

I am going to have to look this up. This is a new one for me.  haha

 

Edit: (time $(i=0; while (( i < 9999999 )); do (( i ++ )); done) - this will return the the time required to crunch the integers between 0 to 9999999) neat!

Edited July 8 by utilitybee

[JorgeB]

Looks like it's not a good test, probably the CPU includes some newer instruction set that makes it much faster, this is usually closer to one minute on older CPUs.

 

I'm sorry but I'm out of ideas, can only tell you that my test server with an inferior CPU, can do 170/180MB/s for a parity check with 30 array devices, so something is likely going on there, but note that my server is also being CPU/memory speed limited, as I'm testing with SSDs and the controllers could do substantially more than that.

 

 

[utilitybee]

Just a thought but does dual parity require more cpu? Just shooting in the dark but if dual parity twice the calculations then my speed would be 1/2 of what you got if you used a single parity. It is pretty close to 1/2 what you got but i do not know what or how you tested. 

 

Even though we did not find the issue. I did learn a few new tricks. So thank you for that.

Edited July 8 by utilitybee

[JorgeB]

1 minute ago, utilitybee said:

Just a thought but does duel parity require more cpu?

Yes, but I tested with dual parity, 30 devices can only be with dual parity.

 

[utilitybee]

I figured you did. haha. Regardless I really do appreciate the time and help. If I figure something out i will comeback and update.

[utilitybee]

Something i thought was interesting is that my array is always capped at this speed. I moved a single large file from the cache to the array with roughly the same speeds.