[SOLVED] Parity check SUPER slow

January 1, 20215 yr

Greetings fellow UnRAIDers, and happy new year!

I'm seeing some very slow parity check speeds (see below) on my monthly parity check today. I did reboot a couple times today after some slowness on the webUI and a docker crash (everything unavailable). The reboot seems to have sorted the docker problem, but now I have a new issue: slow parity speed. It's max/min is between ~25MB/sec and under 1MB/sec, which is far slower than normal.

Enclosed are diagnostics taken during the slow parity check, and a couple screenshots. Any help is very much appreciated. Cheers!

EDIT: Also should mention that I am seeing sustained high CPU usage on the dashboard page (yet not when using htop). Several cores at 100%, overall between 30 and 80% cpu load. Not sure if this is related or not...

Edited January 2, 20215 yr by mattyx

Quote

January 1, 20215 yr

Are all your docker apps working properly? It looks like one keeps restarting over and over again.

Quote

January 1, 20215 yr

Author

They seem to be... All are reachable by web.

Any idea how I can I figure out which docker is doing that? (And predictable follow up question): Any suggestions for how to solve it?

EDIT: As I sent this, Sonarr went unavailable and seems to be unable to be stopped by the webgui. I think maybe thats the one...

Thanks, Squid!

Edited January 1, 20215 yr by mattyx

Quote

January 1, 20215 yr

Author

I've just deleted my docker.img, reinstalled dockers, performed a clean restart, but the parity issue seems to remain...

Quote

January 2, 20215 yr

Community Expert

Please post output of:

sar -dp 5 5

Quote

January 2, 20215 yr

Author

Hey Jorge,

I dont seem to have the system report binary installed, nor do I see it in nerd pack. Any ideas?

Quote

January 2, 20215 yr

Author

Update: Docker rebuild seems not to have solved it. I did notice that my Arq backup to unraid was doing a cleanup task, so I paused that. Since doing that speeds have gone from 1MB to ~20MB, still well below the normal ~100MB average.

Quote

January 2, 20215 yr

Community Expert

Sorry, first install the systat package from nerd pack.

Quote

January 2, 20215 yr

Author

Got it, thanks! Here's the output:

# sar -dp 5 5
Linux 4.19.107-Unraid (Tower) 01/02/2021 _x86_64_ (6 CPU)

11:54:41 AM DEV tps rkB/s wkB/s dkB/s areq-sz aqu-sz await %util
11:54:46 AM loop0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
11:54:46 AM loop1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
11:54:46 AM loop2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
11:54:46 AM loop3 0.20 0.00 0.80 0.00 4.00 0.00 0.00 0.00
11:54:46 AM sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
11:54:46 AM nvme0n1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
11:54:46 AM sdb 8.80 2011.20 31.20 0.00 232.09 0.10 72.50 9.84
11:54:46 AM sdc 6.00 2011.20 4.00 0.00 335.87 0.04 68.93 3.72
11:54:46 AM sdd 5.20 2011.20 0.00 0.00 386.77 0.01 9.81 1.26
11:54:46 AM sdg 7.80 2011.20 16.00 0.00 259.90 0.01 1.18 0.58
11:54:46 AM sdh 6.60 2011.20 4.00 0.00 305.33 0.01 23.30 0.92
11:54:46 AM sdi 5.80 2011.20 0.00 0.00 346.76 0.01 1.97 0.84
11:54:46 AM sdj 7.00 2011.20 8.00 0.00 288.46 0.01 1.23 0.58
11:54:46 AM sde 5.00 2011.20 0.00 0.00 402.24 0.01 3.08 1.02
11:54:46 AM sdf 8.40 2017.60 35.20 0.00 244.38 0.29 403.21 29.50
11:54:46 AM md1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
11:54:46 AM md2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
11:54:46 AM md4 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
11:54:46 AM md5 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
11:54:46 AM md6 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
11:54:46 AM md7 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
11:54:46 AM md9 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
11:54:46 AM md1 0.00 0.00 0.00 0.00 0.00 155.03 0.00 100.02
11:54:46 AM md2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
11:54:46 AM md4 0.00 0.00 0.00 0.00 0.00 8.00 0.00 100.02
11:54:46 AM md5 0.00 0.00 0.00 0.00 0.00 8.00 0.00 100.02
11:54:46 AM md6 0.20 0.00 0.20 0.00 1.00 0.35 0.00 34.60
11:54:46 AM md7 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
11:54:46 AM md9 0.00 0.00 0.00 0.00 0.00 362.07 0.00 100.02
11:54:46 AM nvme0n1p1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

11:54:46 AM DEV tps rkB/s wkB/s dkB/s areq-sz aqu-sz await %util
11:54:51 AM loop0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
11:54:51 AM loop1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
11:54:51 AM loop2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
11:54:51 AM loop3 0.20 0.00 0.80 0.00 4.00 0.00 0.00 0.00
11:54:51 AM sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
11:54:51 AM nvme0n1 0.20 0.00 0.80 0.00 4.00 0.12 0.00 12.48
11:54:51 AM sdb 41.00 6505.60 164.80 0.00 162.69 0.26 56.82 25.66
11:54:51 AM sdc 24.00 6484.00 23.20 0.00 271.13 0.16 24.77 15.56
11:54:51 AM sdd 19.80 6630.40 1.60 0.00 334.95 0.04 3.13 3.86
11:54:51 AM sdg 27.20 6628.80 49.60 0.00 245.53 0.02 1.09 1.98
11:54:51 AM sdh 25.00 6648.00 23.20 0.00 266.85 0.04 10.62 3.42
11:54:51 AM sdi 19.00 6628.80 0.00 0.00 348.88 0.04 3.11 3.68
11:54:51 AM sdj 29.00 6629.60 70.40 0.00 231.03 0.02 1.08 2.12
11:54:51 AM sde 19.00 6628.80 0.00 0.00 348.88 0.04 3.21 3.80
11:54:51 AM sdf 43.60 6669.60 168.00 0.00 156.83 0.24 85.37 23.62
11:54:51 AM md1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
11:54:51 AM md2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
11:54:51 AM md4 0.80 0.00 25.60 0.00 32.00 0.00 0.00 0.00
11:54:51 AM md5 0.60 0.00 19.20 0.00 32.00 0.00 0.00 0.00
11:54:51 AM md6 0.40 0.00 0.30 0.00 0.75 0.00 0.00 0.00
11:54:51 AM md7 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
11:54:51 AM md9 2.80 0.00 42.40 0.00 15.14 0.00 0.00 0.00
11:54:51 AM md1 0.00 0.00 0.00 0.00 0.00 155.00 0.00 100.00
11:54:51 AM md2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
11:54:51 AM md4 1.40 0.00 44.80 0.00 32.00 8.00 22283.00 100.00
11:54:51 AM md5 1.20 0.00 38.40 0.00 32.00 8.00 19750.67 100.00
11:54:51 AM md6 0.40 0.00 0.30 0.00 0.75 0.24 1471.50 24.26
11:54:51 AM md7 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
11:54:51 AM md9 0.00 0.00 0.00 0.00 0.00 362.00 0.00 100.00
11:54:51 AM nvme0n1p1 0.20 0.00 0.80 0.00 4.00 0.00 0.00 0.00

11:54:51 AM DEV tps rkB/s wkB/s dkB/s areq-sz aqu-sz await %util
11:54:56 AM loop0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
11:54:56 AM loop1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
11:54:56 AM loop2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
11:54:56 AM loop3 5.00 0.00 56.00 0.00 11.20 0.00 0.20 0.00
11:54:56 AM sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
11:54:56 AM nvme0n1 5.40 0.00 59.30 0.00 10.98 0.14 0.11 14.14
11:54:56 AM sdb 58.80 6160.80 184.00 0.00 107.90 0.36 117.16 35.94
11:54:56 AM sdc 22.80 6106.40 24.00 0.00 268.88 0.07 20.24 7.24
11:54:56 AM sdd 15.40 5916.80 0.00 0.00 384.21 0.03 2.68 2.80
11:54:56 AM sdg 32.00 5823.20 72.00 0.00 184.22 0.09 10.93 8.58
11:54:56 AM sdh 25.40 5961.60 19.20 0.00 235.46 0.06 34.20 5.88
11:54:56 AM sdi 15.20 5916.80 0.00 0.00 389.26 0.03 2.53 2.90
11:54:56 AM sdj 37.80 6079.20 65.60 0.00 162.56 0.13 93.25 12.82
11:54:56 AM sde 16.20 5928.80 0.00 0.00 365.98 0.05 4.85 5.18
11:54:56 AM sdf 58.80 6154.40 184.00 0.00 107.80 0.41 135.38 41.06
11:54:56 AM md1 11.60 32.00 65.90 0.00 8.44 0.00 0.00 0.00
11:54:56 AM md2 1.20 12.00 0.00 0.00 10.00 0.00 0.00 0.00
11:54:56 AM md4 5.00 25.60 12.80 0.00 7.68 0.00 0.00 0.00
11:54:56 AM md5 0.80 0.00 25.60 0.00 32.00 0.00 0.00 0.00
11:54:56 AM md6 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
11:54:56 AM md7 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
11:54:56 AM md9 14.20 0.00 162.40 0.00 11.44 0.00 0.00 0.00
11:54:56 AM md1 2.00 32.00 0.00 0.00 16.00 101.57 1322202.80 100.00
11:54:56 AM md2 1.20 12.00 0.00 0.00 10.00 0.01 12.50 1.50
11:54:56 AM md4 5.60 25.60 32.00 0.00 10.29 8.32 2425.25 100.00
11:54:56 AM md5 1.60 0.00 51.20 0.00 32.00 8.00 14493.62 100.00
11:54:56 AM md6 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
11:54:56 AM md7 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
11:54:56 AM md9 0.00 0.00 0.00 0.00 0.00 346.78 0.00 100.00
11:54:56 AM nvme0n1p1 5.40 0.00 61.80 0.00 11.44 0.00 0.19 0.10

11:54:56 AM DEV tps rkB/s wkB/s dkB/s areq-sz aqu-sz await %util
11:55:01 AM loop0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
11:55:01 AM loop1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
11:55:01 AM loop2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
11:55:01 AM loop3 0.20 0.00 0.80 0.00 4.00 0.00 0.00 0.00
11:55:01 AM sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
11:55:01 AM nvme0n1 6.20 0.00 25.30 0.00 4.08 0.24 0.16 24.18
11:55:01 AM sdb 32.00 4322.40 112.80 0.00 138.60 0.15 77.39 14.82
11:55:01 AM sdc 15.40 4113.60 17.60 0.00 268.26 0.04 28.73 3.76
11:55:01 AM sdd 10.60 4100.80 0.00 0.00 386.87 0.06 7.79 5.98
11:55:01 AM sdg 18.80 4296.00 20.00 0.00 229.57 0.14 68.47 14.30
11:55:01 AM sdh 21.00 4169.60 16.80 0.00 199.35 0.05 16.46 5.14
11:55:01 AM sdi 10.60 4100.80 0.00 0.00 386.87 0.02 2.83 2.30
11:55:01 AM sdj 18.00 4100.80 58.40 0.00 231.07 0.01 1.07 1.34
11:55:01 AM sde 10.60 4100.80 0.00 0.00 386.87 0.02 3.36 2.44
11:55:01 AM sdf 32.40 4164.80 109.60 0.00 131.93 0.42 403.14 42.40
11:55:01 AM md1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
11:55:01 AM md2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
11:55:01 AM md4 6.20 56.00 19.20 0.00 12.13 0.00 0.00 0.00
11:55:01 AM md5 0.40 0.00 12.80 0.00 32.00 0.00 0.00 0.00
11:55:01 AM md6 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
11:55:01 AM md7 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
11:55:01 AM md9 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
11:55:01 AM md1 0.00 0.00 0.00 0.00 0.00 29.00 0.00 100.00
11:55:01 AM md2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
11:55:01 AM md4 6.60 56.00 32.00 0.00 13.33 9.00 1915.21 100.00
11:55:01 AM md5 0.80 0.00 25.60 0.00 32.00 8.00 8086.00 100.00
11:55:01 AM md6 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
11:55:01 AM md7 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
11:55:01 AM md9 0.20 0.00 3.20 0.00 16.00 288.68 0.00 100.00
11:55:01 AM nvme0n1p1 6.20 0.00 31.40 0.00 5.06 0.00 0.13 0.08

11:55:01 AM DEV tps rkB/s wkB/s dkB/s areq-sz aqu-sz await %util
11:55:06 AM loop0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
11:55:06 AM loop1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
11:55:06 AM loop2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
11:55:06 AM loop3 0.20 0.00 0.80 0.00 4.00 0.00 0.00 0.00
11:55:06 AM sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
11:55:06 AM nvme0n1 6.80 0.00 43.10 0.00 6.34 0.08 0.09 7.66
11:55:06 AM sdb 35.60 5771.20 124.80 0.00 165.62 0.25 49.19 25.02
11:55:06 AM sdc 30.40 5803.20 17.60 0.00 191.47 0.16 22.88 16.42
11:55:06 AM sdd 19.00 5752.80 0.00 0.00 302.78 0.06 4.33 6.38
11:55:06 AM sdg 18.80 5699.20 24.00 0.00 304.43 0.03 2.67 2.60
11:55:06 AM sdh 22.20 5741.60 18.40 0.00 259.46 0.08 13.93 8.06
11:55:06 AM sdi 17.80 5738.40 0.00 0.00 322.38 0.05 3.51 5.22
11:55:06 AM sdj 25.60 5734.40 64.00 0.00 226.50 0.04 11.91 4.40
11:55:06 AM sde 14.60 5693.60 0.00 0.00 389.97 0.03 3.36 3.24
11:55:06 AM sdf 34.20 5730.40 124.00 0.00 171.18 0.35 93.43 34.92
11:55:06 AM md1 0.60 0.80 4.10 0.00 8.17 0.00 0.00 0.00
11:55:06 AM md2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
11:55:06 AM md4 3.00 29.60 12.80 0.00 14.13 0.00 0.00 0.00
11:55:06 AM md5 11.60 96.80 19.20 0.00 10.00 0.00 0.00 0.00
11:55:06 AM md6 4.60 59.20 0.00 0.00 12.87 0.00 0.00 0.00
11:55:06 AM md7 3.40 44.80 0.00 0.00 13.18 0.00 0.00 0.00
11:55:06 AM md9 0.40 0.80 3.20 0.00 10.00 0.00 0.00 0.00
11:55:06 AM md1 0.80 0.80 5.00 0.00 7.25 7.58 487712.50 36.32
11:55:06 AM md2 0.20 0.00 0.20 0.00 1.00 0.15 0.00 14.76
11:55:06 AM md4 3.60 29.60 32.00 0.00 17.11 8.55 1527.72 100.00
11:55:06 AM md5 12.00 96.80 32.00 0.00 10.73 9.18 750.15 100.00
11:55:06 AM md6 4.60 59.20 0.00 0.00 12.87 0.04 7.61 3.34
11:55:06 AM md7 3.40 44.80 0.00 0.00 13.18 0.03 8.71 2.90
11:55:06 AM md9 0.20 0.80 0.00 0.00 4.00 288.34 34519.00 100.00
11:55:06 AM nvme0n1p1 7.00 0.00 43.40 0.00 6.20 0.00 0.03 0.02

Average: DEV tps rkB/s wkB/s dkB/s areq-sz aqu-sz await %util
Average: loop0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Average: loop1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Average: loop2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Average: loop3 1.16 0.00 11.84 0.00 10.21 0.00 0.17 0.00
Average: sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Average: nvme0n1 3.72 0.00 25.70 0.00 6.91 0.12 0.12 11.69
Average: sdb 35.24 4954.24 123.52 0.00 144.09 0.22 79.93 22.26
Average: sdc 19.72 4903.68 17.28 0.00 249.54 0.09 26.44 9.34
Average: sdd 14.00 4882.40 0.32 0.00 348.77 0.04 4.56 4.06
Average: sdg 20.92 4891.68 36.32 0.00 235.56 0.06 16.50 5.61
Average: sdh 20.04 4906.40 16.32 0.00 245.64 0.05 19.39 4.68
Average: sdi 13.68 4879.20 0.00 0.00 356.67 0.03 2.94 2.99
Average: sdj 23.48 4911.04 53.28 0.00 211.43 0.04 33.12 4.25
Average: sde 13.08 4872.64 0.00 0.00 372.53 0.03 3.66 3.14
Average: sdf 35.48 4947.36 124.16 0.00 142.94 0.35 176.59 34.30
Average: md1 2.44 6.56 14.00 0.00 8.43 0.00 0.00 0.00
Average: md2 0.24 2.40 0.00 0.00 10.00 0.00 0.00 0.00
Average: md4 3.00 22.24 14.08 0.00 12.11 0.00 0.00 0.00
Average: md5 2.68 19.36 15.36 0.00 12.96 0.00 0.00 0.00
Average: md6 1.00 11.84 0.06 0.00 11.90 0.00 0.00 0.00
Average: md7 0.68 8.96 0.00 0.00 13.18 0.00 0.00 0.00
Average: md9 3.48 0.16 41.60 0.00 12.00 0.00 0.00 0.00
Average: md1 0.56 6.56 1.00 0.00 13.50 89.63 1083777.00 87.27
Average: md2 0.28 2.40 0.04 0.00 8.71 0.03 10.71 3.25
Average: md4 3.44 22.24 28.16 0.00 14.65 8.37 3658.01 100.00
Average: md5 3.12 19.36 29.44 0.00 15.64 8.24 3997.51 100.00
Average: md6 1.04 11.84 0.10 0.00 11.48 0.12 119.92 12.44
Average: md7 0.68 8.96 0.00 0.00 13.18 0.01 8.71 0.58
Average: md9 0.08 0.16 0.64 0.00 10.00 329.58 3027542.00 100.00
Average: nvme0n1p1 3.76 0.00 27.48 0.00 7.31 0.00 0.11 0.04

Quote

January 2, 20215 yr

Community Expert

Kind of strange results, don't usually see usage on the MD devices, you have more than 1 at 100%, all SMR disks, but not all SMR disks are the same, server load is also very high, likely due to i/o wait, pause the check and run the diskspeed docker.

Quote

January 2, 20215 yr

Community Expert

11:55:06 AM       md1      0.80      0.80      5.00      0.00      7.25      7.58 487712.50     36.32
11:55:06 AM       md2      0.20      0.00      0.20      0.00      1.00      0.15      0.00     14.76
11:55:06 AM       md4      3.60     29.60     32.00      0.00     17.11      8.55   1527.72    100.00
11:55:06 AM       md5     12.00     96.80     32.00      0.00     10.73      9.18    750.15    100.00
11:55:06 AM       md6      4.60     59.20      0.00      0.00     12.87      0.04      7.61      3.34
11:55:06 AM       md7      3.40     44.80      0.00      0.00     13.18      0.03      8.71      2.90
11:55:06 AM       md9      0.20      0.80      0.00      0.00      4.00    288.34  34519.00    100.00

These suggest some activity going on to the array besides the parity check, as that should no register on the md devices, especially disks 4, 5 and 9.

Quote

January 2, 20215 yr

Author

Oddly, after crossing about 5.5% parity check completion, the parity speed has returned to normal levels (currently ~142MB/sec). I am also seeing more normal CPU utilization (~20%, no cores maxed). I'm not sure why, but I'll take it. I'll keep monitoring this for a few more hours, and will mark this solved if the slow parity issue doesn't return.

In the meantime, if there are any other issues I should look into, I welcome the pointers.

Thanks to Squid and JorgeB for the help!!

M

Quote

January 2, 20215 yr

Community Expert

Almost certainly there was something else, like a docker or plugin, using the array.

Quote

January 2, 20215 yr

Author

Got it. Thanks again for the help!

Quote

[SOLVED] Parity check SUPER slow

Featured Replies

Archived

Account

Navigation

Search

Configure browser push notifications

Chrome (Android)

Chrome (Desktop)

Safari (iOS 16.4+)

Safari (macOS)

Edge (Android)

Edge (Desktop)

Firefox (Android)

Firefox (Desktop)