[SOLVED] Parity check SUPER slow


Recommended Posts

Greetings fellow UnRAIDers, and happy new year!

 

I'm seeing some very slow parity check speeds (see below) on my monthly parity check today.  I did reboot a couple times today after some slowness on the webUI and a docker crash (everything unavailable).  The reboot seems to have sorted the docker problem, but now I have a new issue: slow parity speed.  It's max/min is between ~25MB/sec and under 1MB/sec, which is far slower than normal.

 

Enclosed are diagnostics taken during the slow parity check, and a couple screenshots.  Any help is very much appreciated.  Cheers!


EDIT:  Also should mention that I am seeing sustained high CPU usage on the dashboard page (yet not when using htop).  Several cores at 100%, overall between 30 and 80% cpu load.  Not sure if this is related or not...

 

Screen Shot 2021-01-01 at 21.59.29.png

 

Screen Shot 2021-01-01 at 21.59.55.png

 

Edited by mattyx
Link to comment
Posted (edited)

They seem to be...  All are reachable by web. 

 

Any idea how I can I figure out which docker is doing that?  (And predictable follow up question):  Any suggestions for how to solve it?


EDIT:  As I sent this, Sonarr went unavailable and seems to be unable to be stopped by the webgui.  I think maybe thats the one...

 

 

Thanks, Squid!

Edited by mattyx
Link to comment

Update:  Docker rebuild seems not to have solved it.  I did notice that my Arq backup to unraid was doing a cleanup task, so I paused that.  Since doing that speeds have gone from 1MB to ~20MB, still well below the normal ~100MB average.

 

 

Link to comment

Got it, thanks!  Here's the output:

 

# sar -dp 5 5
Linux 4.19.107-Unraid (Tower)     01/02/2021     _x86_64_    (6 CPU)

11:54:41 AM       DEV       tps     rkB/s     wkB/s     dkB/s   areq-sz    aqu-sz     await     %util
11:54:46 AM     loop0      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
11:54:46 AM     loop1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
11:54:46 AM     loop2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
11:54:46 AM     loop3      0.20      0.00      0.80      0.00      4.00      0.00      0.00      0.00
11:54:46 AM       sda      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
11:54:46 AM   nvme0n1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
11:54:46 AM       sdb      8.80   2011.20     31.20      0.00    232.09      0.10     72.50      9.84
11:54:46 AM       sdc      6.00   2011.20      4.00      0.00    335.87      0.04     68.93      3.72
11:54:46 AM       sdd      5.20   2011.20      0.00      0.00    386.77      0.01      9.81      1.26
11:54:46 AM       sdg      7.80   2011.20     16.00      0.00    259.90      0.01      1.18      0.58
11:54:46 AM       sdh      6.60   2011.20      4.00      0.00    305.33      0.01     23.30      0.92
11:54:46 AM       sdi      5.80   2011.20      0.00      0.00    346.76      0.01      1.97      0.84
11:54:46 AM       sdj      7.00   2011.20      8.00      0.00    288.46      0.01      1.23      0.58
11:54:46 AM       sde      5.00   2011.20      0.00      0.00    402.24      0.01      3.08      1.02
11:54:46 AM       sdf      8.40   2017.60     35.20      0.00    244.38      0.29    403.21     29.50
11:54:46 AM       md1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
11:54:46 AM       md2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
11:54:46 AM       md4      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
11:54:46 AM       md5      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
11:54:46 AM       md6      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
11:54:46 AM       md7      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
11:54:46 AM       md9      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
11:54:46 AM       md1      0.00      0.00      0.00      0.00      0.00    155.03      0.00    100.02
11:54:46 AM       md2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
11:54:46 AM       md4      0.00      0.00      0.00      0.00      0.00      8.00      0.00    100.02
11:54:46 AM       md5      0.00      0.00      0.00      0.00      0.00      8.00      0.00    100.02
11:54:46 AM       md6      0.20      0.00      0.20      0.00      1.00      0.35      0.00     34.60
11:54:46 AM       md7      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
11:54:46 AM       md9      0.00      0.00      0.00      0.00      0.00    362.07      0.00    100.02
11:54:46 AM nvme0n1p1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00

11:54:46 AM       DEV       tps     rkB/s     wkB/s     dkB/s   areq-sz    aqu-sz     await     %util
11:54:51 AM     loop0      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
11:54:51 AM     loop1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
11:54:51 AM     loop2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
11:54:51 AM     loop3      0.20      0.00      0.80      0.00      4.00      0.00      0.00      0.00
11:54:51 AM       sda      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
11:54:51 AM   nvme0n1      0.20      0.00      0.80      0.00      4.00      0.12      0.00     12.48
11:54:51 AM       sdb     41.00   6505.60    164.80      0.00    162.69      0.26     56.82     25.66
11:54:51 AM       sdc     24.00   6484.00     23.20      0.00    271.13      0.16     24.77     15.56
11:54:51 AM       sdd     19.80   6630.40      1.60      0.00    334.95      0.04      3.13      3.86
11:54:51 AM       sdg     27.20   6628.80     49.60      0.00    245.53      0.02      1.09      1.98
11:54:51 AM       sdh     25.00   6648.00     23.20      0.00    266.85      0.04     10.62      3.42
11:54:51 AM       sdi     19.00   6628.80      0.00      0.00    348.88      0.04      3.11      3.68
11:54:51 AM       sdj     29.00   6629.60     70.40      0.00    231.03      0.02      1.08      2.12
11:54:51 AM       sde     19.00   6628.80      0.00      0.00    348.88      0.04      3.21      3.80
11:54:51 AM       sdf     43.60   6669.60    168.00      0.00    156.83      0.24     85.37     23.62
11:54:51 AM       md1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
11:54:51 AM       md2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
11:54:51 AM       md4      0.80      0.00     25.60      0.00     32.00      0.00      0.00      0.00
11:54:51 AM       md5      0.60      0.00     19.20      0.00     32.00      0.00      0.00      0.00
11:54:51 AM       md6      0.40      0.00      0.30      0.00      0.75      0.00      0.00      0.00
11:54:51 AM       md7      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
11:54:51 AM       md9      2.80      0.00     42.40      0.00     15.14      0.00      0.00      0.00
11:54:51 AM       md1      0.00      0.00      0.00      0.00      0.00    155.00      0.00    100.00
11:54:51 AM       md2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
11:54:51 AM       md4      1.40      0.00     44.80      0.00     32.00      8.00  22283.00    100.00
11:54:51 AM       md5      1.20      0.00     38.40      0.00     32.00      8.00  19750.67    100.00
11:54:51 AM       md6      0.40      0.00      0.30      0.00      0.75      0.24   1471.50     24.26
11:54:51 AM       md7      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
11:54:51 AM       md9      0.00      0.00      0.00      0.00      0.00    362.00      0.00    100.00
11:54:51 AM nvme0n1p1      0.20      0.00      0.80      0.00      4.00      0.00      0.00      0.00

11:54:51 AM       DEV       tps     rkB/s     wkB/s     dkB/s   areq-sz    aqu-sz     await     %util
11:54:56 AM     loop0      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
11:54:56 AM     loop1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
11:54:56 AM     loop2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
11:54:56 AM     loop3      5.00      0.00     56.00      0.00     11.20      0.00      0.20      0.00
11:54:56 AM       sda      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
11:54:56 AM   nvme0n1      5.40      0.00     59.30      0.00     10.98      0.14      0.11     14.14
11:54:56 AM       sdb     58.80   6160.80    184.00      0.00    107.90      0.36    117.16     35.94
11:54:56 AM       sdc     22.80   6106.40     24.00      0.00    268.88      0.07     20.24      7.24
11:54:56 AM       sdd     15.40   5916.80      0.00      0.00    384.21      0.03      2.68      2.80
11:54:56 AM       sdg     32.00   5823.20     72.00      0.00    184.22      0.09     10.93      8.58
11:54:56 AM       sdh     25.40   5961.60     19.20      0.00    235.46      0.06     34.20      5.88
11:54:56 AM       sdi     15.20   5916.80      0.00      0.00    389.26      0.03      2.53      2.90
11:54:56 AM       sdj     37.80   6079.20     65.60      0.00    162.56      0.13     93.25     12.82
11:54:56 AM       sde     16.20   5928.80      0.00      0.00    365.98      0.05      4.85      5.18
11:54:56 AM       sdf     58.80   6154.40    184.00      0.00    107.80      0.41    135.38     41.06
11:54:56 AM       md1     11.60     32.00     65.90      0.00      8.44      0.00      0.00      0.00
11:54:56 AM       md2      1.20     12.00      0.00      0.00     10.00      0.00      0.00      0.00
11:54:56 AM       md4      5.00     25.60     12.80      0.00      7.68      0.00      0.00      0.00
11:54:56 AM       md5      0.80      0.00     25.60      0.00     32.00      0.00      0.00      0.00
11:54:56 AM       md6      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
11:54:56 AM       md7      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
11:54:56 AM       md9     14.20      0.00    162.40      0.00     11.44      0.00      0.00      0.00
11:54:56 AM       md1      2.00     32.00      0.00      0.00     16.00    101.57 1322202.80    100.00
11:54:56 AM       md2      1.20     12.00      0.00      0.00     10.00      0.01     12.50      1.50
11:54:56 AM       md4      5.60     25.60     32.00      0.00     10.29      8.32   2425.25    100.00
11:54:56 AM       md5      1.60      0.00     51.20      0.00     32.00      8.00  14493.62    100.00
11:54:56 AM       md6      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
11:54:56 AM       md7      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
11:54:56 AM       md9      0.00      0.00      0.00      0.00      0.00    346.78      0.00    100.00
11:54:56 AM nvme0n1p1      5.40      0.00     61.80      0.00     11.44      0.00      0.19      0.10

11:54:56 AM       DEV       tps     rkB/s     wkB/s     dkB/s   areq-sz    aqu-sz     await     %util
11:55:01 AM     loop0      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
11:55:01 AM     loop1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
11:55:01 AM     loop2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
11:55:01 AM     loop3      0.20      0.00      0.80      0.00      4.00      0.00      0.00      0.00
11:55:01 AM       sda      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
11:55:01 AM   nvme0n1      6.20      0.00     25.30      0.00      4.08      0.24      0.16     24.18
11:55:01 AM       sdb     32.00   4322.40    112.80      0.00    138.60      0.15     77.39     14.82
11:55:01 AM       sdc     15.40   4113.60     17.60      0.00    268.26      0.04     28.73      3.76
11:55:01 AM       sdd     10.60   4100.80      0.00      0.00    386.87      0.06      7.79      5.98
11:55:01 AM       sdg     18.80   4296.00     20.00      0.00    229.57      0.14     68.47     14.30
11:55:01 AM       sdh     21.00   4169.60     16.80      0.00    199.35      0.05     16.46      5.14
11:55:01 AM       sdi     10.60   4100.80      0.00      0.00    386.87      0.02      2.83      2.30
11:55:01 AM       sdj     18.00   4100.80     58.40      0.00    231.07      0.01      1.07      1.34
11:55:01 AM       sde     10.60   4100.80      0.00      0.00    386.87      0.02      3.36      2.44
11:55:01 AM       sdf     32.40   4164.80    109.60      0.00    131.93      0.42    403.14     42.40
11:55:01 AM       md1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
11:55:01 AM       md2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
11:55:01 AM       md4      6.20     56.00     19.20      0.00     12.13      0.00      0.00      0.00
11:55:01 AM       md5      0.40      0.00     12.80      0.00     32.00      0.00      0.00      0.00
11:55:01 AM       md6      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
11:55:01 AM       md7      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
11:55:01 AM       md9      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
11:55:01 AM       md1      0.00      0.00      0.00      0.00      0.00     29.00      0.00    100.00
11:55:01 AM       md2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
11:55:01 AM       md4      6.60     56.00     32.00      0.00     13.33      9.00   1915.21    100.00
11:55:01 AM       md5      0.80      0.00     25.60      0.00     32.00      8.00   8086.00    100.00
11:55:01 AM       md6      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
11:55:01 AM       md7      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
11:55:01 AM       md9      0.20      0.00      3.20      0.00     16.00    288.68      0.00    100.00
11:55:01 AM nvme0n1p1      6.20      0.00     31.40      0.00      5.06      0.00      0.13      0.08

11:55:01 AM       DEV       tps     rkB/s     wkB/s     dkB/s   areq-sz    aqu-sz     await     %util
11:55:06 AM     loop0      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
11:55:06 AM     loop1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
11:55:06 AM     loop2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
11:55:06 AM     loop3      0.20      0.00      0.80      0.00      4.00      0.00      0.00      0.00
11:55:06 AM       sda      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
11:55:06 AM   nvme0n1      6.80      0.00     43.10      0.00      6.34      0.08      0.09      7.66
11:55:06 AM       sdb     35.60   5771.20    124.80      0.00    165.62      0.25     49.19     25.02
11:55:06 AM       sdc     30.40   5803.20     17.60      0.00    191.47      0.16     22.88     16.42
11:55:06 AM       sdd     19.00   5752.80      0.00      0.00    302.78      0.06      4.33      6.38
11:55:06 AM       sdg     18.80   5699.20     24.00      0.00    304.43      0.03      2.67      2.60
11:55:06 AM       sdh     22.20   5741.60     18.40      0.00    259.46      0.08     13.93      8.06
11:55:06 AM       sdi     17.80   5738.40      0.00      0.00    322.38      0.05      3.51      5.22
11:55:06 AM       sdj     25.60   5734.40     64.00      0.00    226.50      0.04     11.91      4.40
11:55:06 AM       sde     14.60   5693.60      0.00      0.00    389.97      0.03      3.36      3.24
11:55:06 AM       sdf     34.20   5730.40    124.00      0.00    171.18      0.35     93.43     34.92
11:55:06 AM       md1      0.60      0.80      4.10      0.00      8.17      0.00      0.00      0.00
11:55:06 AM       md2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
11:55:06 AM       md4      3.00     29.60     12.80      0.00     14.13      0.00      0.00      0.00
11:55:06 AM       md5     11.60     96.80     19.20      0.00     10.00      0.00      0.00      0.00
11:55:06 AM       md6      4.60     59.20      0.00      0.00     12.87      0.00      0.00      0.00
11:55:06 AM       md7      3.40     44.80      0.00      0.00     13.18      0.00      0.00      0.00
11:55:06 AM       md9      0.40      0.80      3.20      0.00     10.00      0.00      0.00      0.00
11:55:06 AM       md1      0.80      0.80      5.00      0.00      7.25      7.58 487712.50     36.32
11:55:06 AM       md2      0.20      0.00      0.20      0.00      1.00      0.15      0.00     14.76
11:55:06 AM       md4      3.60     29.60     32.00      0.00     17.11      8.55   1527.72    100.00
11:55:06 AM       md5     12.00     96.80     32.00      0.00     10.73      9.18    750.15    100.00
11:55:06 AM       md6      4.60     59.20      0.00      0.00     12.87      0.04      7.61      3.34
11:55:06 AM       md7      3.40     44.80      0.00      0.00     13.18      0.03      8.71      2.90
11:55:06 AM       md9      0.20      0.80      0.00      0.00      4.00    288.34  34519.00    100.00
11:55:06 AM nvme0n1p1      7.00      0.00     43.40      0.00      6.20      0.00      0.03      0.02

Average:          DEV       tps     rkB/s     wkB/s     dkB/s   areq-sz    aqu-sz     await     %util
Average:        loop0      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
Average:        loop1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
Average:        loop2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
Average:        loop3      1.16      0.00     11.84      0.00     10.21      0.00      0.17      0.00
Average:          sda      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
Average:      nvme0n1      3.72      0.00     25.70      0.00      6.91      0.12      0.12     11.69
Average:          sdb     35.24   4954.24    123.52      0.00    144.09      0.22     79.93     22.26
Average:          sdc     19.72   4903.68     17.28      0.00    249.54      0.09     26.44      9.34
Average:          sdd     14.00   4882.40      0.32      0.00    348.77      0.04      4.56      4.06
Average:          sdg     20.92   4891.68     36.32      0.00    235.56      0.06     16.50      5.61
Average:          sdh     20.04   4906.40     16.32      0.00    245.64      0.05     19.39      4.68
Average:          sdi     13.68   4879.20      0.00      0.00    356.67      0.03      2.94      2.99
Average:          sdj     23.48   4911.04     53.28      0.00    211.43      0.04     33.12      4.25
Average:          sde     13.08   4872.64      0.00      0.00    372.53      0.03      3.66      3.14
Average:          sdf     35.48   4947.36    124.16      0.00    142.94      0.35    176.59     34.30
Average:          md1      2.44      6.56     14.00      0.00      8.43      0.00      0.00      0.00
Average:          md2      0.24      2.40      0.00      0.00     10.00      0.00      0.00      0.00
Average:          md4      3.00     22.24     14.08      0.00     12.11      0.00      0.00      0.00
Average:          md5      2.68     19.36     15.36      0.00     12.96      0.00      0.00      0.00
Average:          md6      1.00     11.84      0.06      0.00     11.90      0.00      0.00      0.00
Average:          md7      0.68      8.96      0.00      0.00     13.18      0.00      0.00      0.00
Average:          md9      3.48      0.16     41.60      0.00     12.00      0.00      0.00      0.00
Average:          md1      0.56      6.56      1.00      0.00     13.50     89.63 1083777.00     87.27
Average:          md2      0.28      2.40      0.04      0.00      8.71      0.03     10.71      3.25
Average:          md4      3.44     22.24     28.16      0.00     14.65      8.37   3658.01    100.00
Average:          md5      3.12     19.36     29.44      0.00     15.64      8.24   3997.51    100.00
Average:          md6      1.04     11.84      0.10      0.00     11.48      0.12    119.92     12.44
Average:          md7      0.68      8.96      0.00      0.00     13.18      0.01      8.71      0.58
Average:          md9      0.08      0.16      0.64      0.00     10.00    329.58 3027542.00    100.00
Average:    nvme0n1p1      3.76      0.00     27.48      0.00      7.31      0.00      0.11      0.04

Link to comment

Kind of strange results, don't usually see usage on the MD devices, you have more than 1 at 100%, all SMR disks, but not all SMR disks are the same, server load is also very high, likely due to i/o wait, pause the check and run the diskspeed docker.

Link to comment
11:55:06 AM       md1      0.80      0.80      5.00      0.00      7.25      7.58 487712.50     36.32
11:55:06 AM       md2      0.20      0.00      0.20      0.00      1.00      0.15      0.00     14.76
11:55:06 AM       md4      3.60     29.60     32.00      0.00     17.11      8.55   1527.72    100.00
11:55:06 AM       md5     12.00     96.80     32.00      0.00     10.73      9.18    750.15    100.00
11:55:06 AM       md6      4.60     59.20      0.00      0.00     12.87      0.04      7.61      3.34
11:55:06 AM       md7      3.40     44.80      0.00      0.00     13.18      0.03      8.71      2.90
11:55:06 AM       md9      0.20      0.80      0.00      0.00      4.00    288.34  34519.00    100.00

 

These suggest some activity going on to the array besides the parity check, as that should no register on the md devices, especially disks 4, 5 and 9.

Link to comment

Oddly, after crossing about 5.5% parity check completion, the parity speed has returned to normal levels (currently ~142MB/sec).  I am also seeing more normal CPU utilization (~20%, no cores maxed).  I'm not sure why, but I'll take it.  :) I'll keep monitoring this for a few more hours, and will mark this solved if the slow parity issue doesn't return. 

 

In the meantime, if there are any other issues I should look into, I welcome the pointers. 

 

Thanks to Squid and JorgeB for the help!!

 

M

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.