Slow Parity Rebuild

[AshranPewter]

Hi All

having some issues building my parity, this isn't writes to the server or anything but just parity rebuild.

It is at 30MB/s, now I wouldn't usually care that much but this machine was a pre-built QNAP that had very fast transfer times and raid rebuild, it seems weird that I'm stuck at 30MB/s when the tests I run on each of the drives is 150MB/s reads (using the drivespeed docker). 2 * 4TB and the parity drive is 8TB.

 

Note: 2 things to note,

1. It transfers faster with files downloading (from a docker container download) getting speeds up to 60MB/s
2. There is no change with docker enabled or disabled

 

Attached is the diagnostics.

 

Thanks!

Ash

tianding-unraid-diagnostics-20201212-2306.zip

[Vr2Io]

Have many abnormal call trace in log. Stress memory test first.

[AshranPewter]

I think that did it, getting more consistent around 70-80 after reseating. Probably faster when Dockers are down but not fussed about it.

Also seeing my processor cores consistently at more utilization whereas before was stuck around 42% overall with one core at 100%. Probably causing issues there. Appreciate the help.

 

Cheers,

Ash

[AshranPewter]

Nevermind.... the traces you mentioned are back and the speeds are back down to 30-40MB/s...

[Vr2Io]

1 minute ago, AshranPewter said:

30-40MB/s...

But speed abnormal, seems you use two port by chipset and two port in Asmedia 1061, those setup was common, speed expect over 130MB/s.

 

Anyway, call trace must be fix.

[Vr2Io]

31 minutes ago, AshranPewter said:

Probably causing issues there. Appreciate the help.

Not likely, my first setup also ATOM grade CPU (J1800 have higher clock rate), even in high utilization speed also ~83MB/s or 110MB/s in sigle parity with ~12 disks.

 

Call trace must fix.

[AshranPewter]

I've run a memtest and no issues 100%, i tried a different set of ram.

Trace doesn't go away.

Not sure what to do.

[Zonediver]

4 hours ago, AshranPewter said:

I've run a memtest and no issues 100%, i tried a different set of ram.

Trace doesn't go away.

Not sure what to do.

Next step: CPU and Mainboard... but this will be hard if you dont have spareparts...

[AshranPewter]

It's a QNAP machine, getting new CPU/Mainboard will be next to impossible. I'll just deal with slow parity rebuilds if there's no other solution.

[AshranPewter]

Found this thread: 

that suggests reducing md_sync_thresh is there somewhere I can change this in the interface? or a place I can see the current value?

 

Thanks!

Ash

 

[JorgeB]

23 minutes ago, AshranPewter said:

that suggests reducing md_sync_thresh

That tunable no longer exists in current Unraid.

[jonp]

Hi there,

 

The call traces definitely look to be pointing as the cause of the problem.  Out of curiosity, can you try upgrading to 6.9-rc1 to see if that changes anything for you?  6.9 is on a newer Linux kernel and I'm curious of those call traces get resolved by upgrading.  You can try this by going to the Tools > Update OS page, selecting "Next" from the drop down there, and then installing RC1 and rebooting your server.  Let us know how that goes and after, please include an updated system diagnostics.

 

All the best,

 

Jon

[AshranPewter]

Thanks for the reply @jonp!

I installed 6.9 rc1 and no change. When doing a parity rebuild I still get the problems before. Max 40MB/s, faster when downloading large files from a program in Docker and the Traces are still there.

 

I did try a few added things to see if I could fix it but neither of them fixed anything.

1. Tried Booting into Safe Mode
2. Tried Booting with Bios instead of UEFI

Additionally I have issues with where the WebGUI is unresponsive when I start the Parity Check

tianding-unraid-diagnostics-20201217-1223.zip

[AshranPewter]

I figured it out.

I had used pci=noacpi to get all drives to be recognized but pcie_port_pm=off is a better option per the the bug report here https://bugzilla.kernel.org/show_bug.cgi?id=196805

[Vr2Io]

Most BIOS should default disable ASPM on PCIe, BIOS no those setting ?

Edited December 17, 2020 by Vr2Io