(Solved) Parity-Sync running extremely slowly...currently at 8MB/sec Unable to find out why.

[FQs19]

After rebuilding my server and setting everything up, I finally added my two WD Red Pro disks back as Parity.

The first time I tried adding the two parity disks, it was running at 10MB/sec. 

I canceled that, rebooted my server and tried again. 

This time the performance is worse. It's currently running at 8MB/sec.

 

I ran all disks through the DiskSpeed docker and all disks looked great. 

I know some disks are old, but I've been able to transfer data to my array at the normal 110-120MB/sec.

 

I canceled the Parity-Sync so I could grab the DiskSpeed graph.

After stopping my array, setting my two parity drives, and starting the array, the speed is still slow at 8.7MB/sec.

 

Anyone have an idea of why the sync is so slow?

 

On version 6.10.3.

All plugins updated. 

Two parity drives are WD Red Pro's.

Array disk are all WD with a mix of red, red plus, and pro.

threadripper19-diagnostics-20220704-2135.zip

Edited July 31, 2022 by FQs19
changed to solved

[Vr2Io]

No sdj ( parity ) result ? Both parity use FCH SATA controller, could you try connect to ASM1062 SATA controller.

[JorgeB]

The sync thread is single threaded and the CPU usage is very high for the speed it's going, there was a similar case recently, also with a threadripper IIRC, but don't remember the solution, let me see if I can find it.

[JorgeB]

Found the other thread but there's no clear solution, some things to try though:

 

https://forums.unraid.net/topic/123903-slow-parity-rebuild/?do=findComment&comment=1130584

 

 

 

[FQs19]

I'm thinking @Vr2Io might be right.

I don't remember how I had my disks connected to my motherboard and HBA. It's been a minute. HAHA

I thought I had 4 disks (including the parity disks) connected to my motherboard's 4 SATA ports with nothing connected to the ASMedia SATA ports and 8 disks connected to my HBA card. 

 

I'll try and move my parity disks to the HBA card and see what happens.

@JorgeB @Vr2Io would either of you suggest maybe spreading the two parity disks over the motherboard ports? Like 1 on the motherboard SATA ports and the other on the ASMedia ports. How would either of you attach 14 drives to an 8 port HBA, 4 MB SATA ports, and 2 ASMedia ports??

 

I forget to mention, I moved my HBA to the first x8 slot from the second x16 slot. Would that make a difference?

I currently have my Quadro RTX4000 on slot 1 (x16), the HBA on slot 2 (x8), slots 3 and 4 are empty. I will eventually put another video card in for a VM, but after I get all this sorted.  

[Vr2Io]

1 hour ago, FQs19 said:

would either of you suggest maybe spreading the two parity disks over the motherboard ports?

In general, I won't cross controller although cross have redundant in controller view.

 

1 hour ago, FQs19 said:

I forget to mention, I moved my HBA to the first x8 slot from the second x16 slot. Would that make a difference?

No difference, as HBA was x8

Edited July 5, 2022 by Vr2Io

[FQs19]

@JorgeB @Vr2Io

 

I moved my parity drives to the ASMedia ports. 

So now I have 8 disks on the HBA, 2 disks on the MB SATA ports, and 2 parity disks on the ASMedia SATA ports. 

 

Speed increased to only 12MB/sec.

 

CPU occasionally has 1 thread hitting 80%, but this is the average:

 

Any other ideas?

threadripper19-diagnostics-20220705-1524.zip

[Vr2Io]

Not much idea, TR is kind of NUMA, memory and IO may access cross CPU core, but those delay definitely neglectable.

 

I have 1920x, not running Unraid and only have three harddisk attach, will do some test.

 

Pls try disable parity2, if that will increase speed then it should CPU issue relateed.

Edited July 5, 2022 by Vr2Io

[FQs19]

6 minutes ago, Vr2Io said:

Not much idea, TR is kind of NUMA, memory and IO may access cross CPU core, but those delay definitely neglectable.

 

I have 1920x, not running Unraid and only have three harddisk attach, will do some test.

 

Pls try disable parity2, if that will increase speed then it should CPU issue relateed.

 

No change with just 1 parity disk:

 

I'm going to Clear my CMOS, start Unraid and try again.

If that doesn't work, I'll move my parity disks to my HBA and try again. 

 

Thanks for the help. 

threadripper19-diagnostics-20220705-1554.zip

[FQs19]

So I cleared the CMOS and rebooted. No change. Still only getting 10MB/sec.

I then moved my parity disks to the HBA. Still no change. Still only getting 10MB/sec, except this time my second parity disk through a UDMA CRC error, well actually two. It was zero now it's 2. 

This motherboard and/or CPU is toast.

There's only one thing left to try and that's to re-flash the MB BIOS back to 1603 which gives me CPU Code 7 and TEST ERRORS when booting if I disable the Intel LAN and WiFi controllers.  I'm currently running a special BIOS that Asus made for me to fix those errors; BIOS 9901. 

So if re-flashing back to 1603 allows parity to sync normally, I'll have to decide whether to continuing using BIOS 1603 dealing with the errors and whatever other anomalies there are or write back to Asus and tell them I can't run a parity sync.

 

This is by far the worst computer I've ever built. I've never had such horrendous luck with pc components like this especially while they're still under warranty with no possibility to get new ones. Biggest waste of $2440 ever. 

threadripper19-diagnostics-20220705-2146.zip

[Vr2Io]

I have make test on my 1920x, can't reproduce the problem. CPU core will reach ceiling 80+ for short time and not in continues.

 

 

Could you do one more test, only test with two parity disk, new config, then assign one as data and other as parity.

 

Edited July 6, 2022 by Vr2Io

[FQs19]

4 minutes ago, Vr2Io said:

I have make test on my 1920x, can't reproduce the problem. CPU core will reach ceiling 80+ for short time and not in continues.

 

 

Could you do one more test, only test with two parity disk, new config, then assign one as data and other as parity.

 

 

I can try that. 

Do I need to disconnect all the other disks or can I just leave them as unassigned ones for this test?

So New Config, one old parity disk as Parity and the other old parity disk as Data Disk 1.

 

Also, what are the steps for getting back to my current assignment? I'm assuming just populate the slots with the screenshot I have now, keeping the parity empty. Is that correct?

[Vr2Io]

1 minute ago, FQs19 said:

Do I need to disconnect all the other disks or can I just leave them as unassigned ones for this test?

So New Config, one old parity disk as Parity and the other old parity disk as Data Disk 1.

Both yes, no need disconnect any disk. ** Just don't wrongly assign real data disk be parity **

 

Below result just for fun, no much different on CPU usage

[FQs19]

Just now, Vr2Io said:

Both yes, no need disconnect any disk. ** Just don't wrongly assign real data disk be parity **

 

Below result just for fun, no much different on CPU usage

I'll give it a try. Thanks.

[FQs19]

12 minutes ago, Vr2Io said:

Both yes, no need disconnect any disk. ** Just don't wrongly assign real data disk be parity **

 

Below result just for fun, no much different on CPU usage

Well it somewhat worked, though I'm only getting around 60MB/sec. It occasionally spikes up into the 80MB/sec range. 

CPU usage is like before:

 

This still isn't anywhere near the 180MB/sec. I should be seeing. 

What are your thoughts on this??

threadripper19-diagnostics-20220705-2234.zip

[Vr2Io]

Good, at least have some different, I notice your 2nd parity disk "Seek time performance" was 15 and I check all my WD disk was 18, it may indicate this disk have issue.

Pls check "Seek time performance" on 1st parity disk. If it is 18 or other value, then it may prove 2nd disk have some issue.

 

Next, I suggest assign a real data disk and 1st parity disk as parity and perform test again.

 

** Just don't wrongly assign any real data disk be parity "

 

I got the value is 15 too, pls in console terminal type "watch -n 1 grep MHz /proc/cpuinfo" to check current CPU frequency.

 

 

Edited July 6, 2022 by Vr2Io

[FQs19]

3 minutes ago, Vr2Io said:

Good, at least have some different, I notice your 2nd parity disk "Seek time performance" was 15 and I check all my WD disk was 18, it may indicate this disk have issue.

Pls check "Seek time performance" on 1st parity disk. If it is 18 or other value, then it may prove 2nd disk have some issue.

 

Next, I suggest assign a real data disk and 1st parity disk as parity and perform test again.

 

** Just don't wrongly assign any real data disk be parity "

 

"Seek time Performance" for both are 15.

 

I'll try again with first Parity as Parity and a Data disk as Disk 1.

[FQs19]

@Vr2Io

 

Same results as first test. ~60MB/sec. with occasional spikes to 80MB/sec.

For the first second after starting the array, I did see a speed of 120MB/sec, but it never went that high again.

[FQs19]

@Vr2Io

I swapped old parity disks and tried again. Same results, only getting ~60MB/sec. 

So disk ...KRD as parity 60MB/sec. and disk ...P4D as parity 60MB/sec. 

Edited July 6, 2022 by FQs19
added screenshot

[Vr2Io]

Pls in console terminal type "watch -n 1 grep MHz /proc/cpuinfo" to check actual CPU frequency.

[FQs19]

I stopped the parity-sync and did a New Config, so this is just unraid sitting idle on the Main page.

Every 1.0s: grep MHz /proc/cpuinfo                                                                     Threadripper19: Tue Jul  5 22:59:38 2022

cpu MHz         : 2918.511
cpu MHz         : 2427.985
cpu MHz         : 2200.000
cpu MHz         : 2800.000
cpu MHz         : 2775.257
cpu MHz         : 2800.000
cpu MHz         : 2800.000
cpu MHz         : 2800.000
cpu MHz         : 2800.000
cpu MHz         : 2800.000
cpu MHz         : 2863.275
cpu MHz         : 2744.736
cpu MHz         : 2200.000
cpu MHz         : 3002.078
cpu MHz         : 2800.000
cpu MHz         : 2800.000
cpu MHz         : 3369.383
cpu MHz         : 2800.000
cpu MHz         : 2514.198
cpu MHz         : 2800.000
cpu MHz         : 2798.982
cpu MHz         : 2800.000
cpu MHz         : 2800.000
cpu MHz         : 2785.893
cpu MHz         : 3800.000
cpu MHz         : 2800.000
cpu MHz         : 2800.000
cpu MHz         : 2865.946
cpu MHz         : 2800.000
cpu MHz         : 2854.917
cpu MHz         : 2800.000
cpu MHz         : 2800.000
cpu MHz         : 2800.000
cpu MHz         : 2800.000
cpu MHz         : 2800.000
cpu MHz         : 2781.481
cpu MHz         : 2800.000
cpu MHz         : 2800.000
cpu MHz         : 2800.000
cpu MHz         : 2800.000
cpu MHz         : 2800.000
cpu MHz         : 3066.897
cpu MHz         : 2856.996
cpu MHz         : 2800.000
cpu MHz         : 2800.000
cpu MHz         : 2929.347
cpu MHz         : 2800.000
cpu MHz         : 2800.000

 

 

If you want to see this while it's running the test, let me know. 

[Vr2Io]

1 hour ago, FQs19 said:

There's only one thing left to try and that's to re-flash the MB BIOS back to 1603 which gives me CPU Code 7 and TEST ERRORS when booting if I disable the Intel LAN and WiFi controllers.  I'm currently running a special BIOS that Asus made for me to fix those errors; BIOS 9901. 

So if re-flashing back to 1603 allows parity to sync normally, I'll have to decide whether to continuing using BIOS 1603 dealing with the errors and whatever other anomalies there are or write back to Asus and tell them I can't run a parity sync.

Thanks, CPU frequency also great, Then I agree you should go above route, good luck !!

 

Or just test two parity disk at different SATA controller instead HBA, one step more.

 

** Post have edit again **

Edited July 6, 2022 by Vr2Io

[FQs19]

Just now, Vr2Io said:

Thanks, CPU frequency also great, Then I agree you should go above route, good luck !!

I figured as much.

It might be a couple days before I can try that, but I'll continue posting when I do. 

Thank you so much for the help.

[FQs19]

@Vr2Io

 

I couldn't sleep without trying the 1603 BIOS.

Performance is better, but not by much. Not sure if this speed is normal, but it's not the 180MB/sec. I'm expecting. 

 

This is one of my old parity disks ...KRD as parity and one of data disks as Disk 1:

 

It was over 100MB/sec for a couple seconds after starting the array, but kept falling to ~80MB and leveled off there. 

Here's the CPU usage as well:

Every 1.0s: grep MHz /proc/cpuinfo                                                                              Threadripper19: Tue Jul  5 23:27:31 2022

cpu MHz         : 2800.000
cpu MHz         : 2845.614
cpu MHz         : 2851.384
cpu MHz         : 2945.021
cpu MHz         : 2800.000
cpu MHz         : 2200.000
cpu MHz         : 2800.000
cpu MHz         : 4241.914
cpu MHz         : 2800.000
cpu MHz         : 2800.000
cpu MHz         : 2462.802
cpu MHz         : 3296.135
cpu MHz         : 2800.000
cpu MHz         : 2905.352
cpu MHz         : 2800.000
cpu MHz         : 2800.000
cpu MHz         : 3259.315
cpu MHz         : 2800.000
cpu MHz         : 2800.000
cpu MHz         : 2967.138
cpu MHz         : 2800.000
cpu MHz         : 3800.000
cpu MHz         : 2342.124
cpu MHz         : 4240.152
cpu MHz         : 2800.000
cpu MHz         : 2889.171
cpu MHz         : 2200.000
cpu MHz         : 2800.000
cpu MHz         : 2716.848
cpu MHz         : 2800.000
cpu MHz         : 2885.461
cpu MHz         : 3800.000
cpu MHz         : 2962.822
cpu MHz         : 2800.000
cpu MHz         : 2800.000
cpu MHz         : 3227.638
cpu MHz         : 2800.000
cpu MHz         : 2800.000
cpu MHz         : 2800.000
cpu MHz         : 2800.000
cpu MHz         : 2200.000
cpu MHz         : 2866.924
cpu MHz         : 2794.674
cpu MHz         : 2800.000
cpu MHz         : 2892.098
cpu MHz         : 2800.000
cpu MHz         : 2347.624
cpu MHz         : 2200.000

 

 

I'm going to try again with just the old parity disks like the first test you asked me to try. 

[Vr2Io]

4 minutes ago, FQs19 said:

Performance is better, but not by much. Not sure if this speed is normal, but it's not the 180MB/sec. I'm expecting. 

Sure abnormal.

 

4 minutes ago, FQs19 said:

I couldn't sleep without trying the 1603 BIOS.

Suggest only use parity disk to test, connect to onboard FCH or Asmedia SATA controller and perform same test, otherwise you really need try different BIOS / Unraid OS to figure out whats going wrong.

Edited July 6, 2022 by Vr2Io