mattyx Posted January 1, 2021 Share Posted January 1, 2021 (edited) Greetings fellow UnRAIDers, and happy new year! I'm seeing some very slow parity check speeds (see below) on my monthly parity check today. I did reboot a couple times today after some slowness on the webUI and a docker crash (everything unavailable). The reboot seems to have sorted the docker problem, but now I have a new issue: slow parity speed. It's max/min is between ~25MB/sec and under 1MB/sec, which is far slower than normal. Enclosed are diagnostics taken during the slow parity check, and a couple screenshots. Any help is very much appreciated. Cheers! EDIT: Also should mention that I am seeing sustained high CPU usage on the dashboard page (yet not when using htop). Several cores at 100%, overall between 30 and 80% cpu load. Not sure if this is related or not... Edited January 2, 2021 by mattyx Quote Link to comment
Squid Posted January 1, 2021 Share Posted January 1, 2021 Are all your docker apps working properly? It looks like one keeps restarting over and over again. Quote Link to comment
mattyx Posted January 1, 2021 Author Share Posted January 1, 2021 (edited) They seem to be... All are reachable by web. Any idea how I can I figure out which docker is doing that? (And predictable follow up question): Any suggestions for how to solve it? EDIT: As I sent this, Sonarr went unavailable and seems to be unable to be stopped by the webgui. I think maybe thats the one... Thanks, Squid! Edited January 1, 2021 by mattyx Quote Link to comment
mattyx Posted January 1, 2021 Author Share Posted January 1, 2021 I've just deleted my docker.img, reinstalled dockers, performed a clean restart, but the parity issue seems to remain... Quote Link to comment
JorgeB Posted January 2, 2021 Share Posted January 2, 2021 Please post output of: sar -dp 5 5 Quote Link to comment
mattyx Posted January 2, 2021 Author Share Posted January 2, 2021 Hey Jorge, I dont seem to have the system report binary installed, nor do I see it in nerd pack. Any ideas? Quote Link to comment
mattyx Posted January 2, 2021 Author Share Posted January 2, 2021 Update: Docker rebuild seems not to have solved it. I did notice that my Arq backup to unraid was doing a cleanup task, so I paused that. Since doing that speeds have gone from 1MB to ~20MB, still well below the normal ~100MB average. Quote Link to comment
JorgeB Posted January 2, 2021 Share Posted January 2, 2021 Sorry, first install the systat package from nerd pack. Quote Link to comment
mattyx Posted January 2, 2021 Author Share Posted January 2, 2021 Got it, thanks! Here's the output: # sar -dp 5 5 Linux 4.19.107-Unraid (Tower) 01/02/2021 _x86_64_ (6 CPU) 11:54:41 AM DEV tps rkB/s wkB/s dkB/s areq-sz aqu-sz await %util 11:54:46 AM loop0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 11:54:46 AM loop1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 11:54:46 AM loop2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 11:54:46 AM loop3 0.20 0.00 0.80 0.00 4.00 0.00 0.00 0.00 11:54:46 AM sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 11:54:46 AM nvme0n1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 11:54:46 AM sdb 8.80 2011.20 31.20 0.00 232.09 0.10 72.50 9.84 11:54:46 AM sdc 6.00 2011.20 4.00 0.00 335.87 0.04 68.93 3.72 11:54:46 AM sdd 5.20 2011.20 0.00 0.00 386.77 0.01 9.81 1.26 11:54:46 AM sdg 7.80 2011.20 16.00 0.00 259.90 0.01 1.18 0.58 11:54:46 AM sdh 6.60 2011.20 4.00 0.00 305.33 0.01 23.30 0.92 11:54:46 AM sdi 5.80 2011.20 0.00 0.00 346.76 0.01 1.97 0.84 11:54:46 AM sdj 7.00 2011.20 8.00 0.00 288.46 0.01 1.23 0.58 11:54:46 AM sde 5.00 2011.20 0.00 0.00 402.24 0.01 3.08 1.02 11:54:46 AM sdf 8.40 2017.60 35.20 0.00 244.38 0.29 403.21 29.50 11:54:46 AM md1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 11:54:46 AM md2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 11:54:46 AM md4 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 11:54:46 AM md5 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 11:54:46 AM md6 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 11:54:46 AM md7 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 11:54:46 AM md9 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 11:54:46 AM md1 0.00 0.00 0.00 0.00 0.00 155.03 0.00 100.02 11:54:46 AM md2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 11:54:46 AM md4 0.00 0.00 0.00 0.00 0.00 8.00 0.00 100.02 11:54:46 AM md5 0.00 0.00 0.00 0.00 0.00 8.00 0.00 100.02 11:54:46 AM md6 0.20 0.00 0.20 0.00 1.00 0.35 0.00 34.60 11:54:46 AM md7 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 11:54:46 AM md9 0.00 0.00 0.00 0.00 0.00 362.07 0.00 100.02 11:54:46 AM nvme0n1p1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 11:54:46 AM DEV tps rkB/s wkB/s dkB/s areq-sz aqu-sz await %util 11:54:51 AM loop0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 11:54:51 AM loop1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 11:54:51 AM loop2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 11:54:51 AM loop3 0.20 0.00 0.80 0.00 4.00 0.00 0.00 0.00 11:54:51 AM sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 11:54:51 AM nvme0n1 0.20 0.00 0.80 0.00 4.00 0.12 0.00 12.48 11:54:51 AM sdb 41.00 6505.60 164.80 0.00 162.69 0.26 56.82 25.66 11:54:51 AM sdc 24.00 6484.00 23.20 0.00 271.13 0.16 24.77 15.56 11:54:51 AM sdd 19.80 6630.40 1.60 0.00 334.95 0.04 3.13 3.86 11:54:51 AM sdg 27.20 6628.80 49.60 0.00 245.53 0.02 1.09 1.98 11:54:51 AM sdh 25.00 6648.00 23.20 0.00 266.85 0.04 10.62 3.42 11:54:51 AM sdi 19.00 6628.80 0.00 0.00 348.88 0.04 3.11 3.68 11:54:51 AM sdj 29.00 6629.60 70.40 0.00 231.03 0.02 1.08 2.12 11:54:51 AM sde 19.00 6628.80 0.00 0.00 348.88 0.04 3.21 3.80 11:54:51 AM sdf 43.60 6669.60 168.00 0.00 156.83 0.24 85.37 23.62 11:54:51 AM md1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 11:54:51 AM md2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 11:54:51 AM md4 0.80 0.00 25.60 0.00 32.00 0.00 0.00 0.00 11:54:51 AM md5 0.60 0.00 19.20 0.00 32.00 0.00 0.00 0.00 11:54:51 AM md6 0.40 0.00 0.30 0.00 0.75 0.00 0.00 0.00 11:54:51 AM md7 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 11:54:51 AM md9 2.80 0.00 42.40 0.00 15.14 0.00 0.00 0.00 11:54:51 AM md1 0.00 0.00 0.00 0.00 0.00 155.00 0.00 100.00 11:54:51 AM md2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 11:54:51 AM md4 1.40 0.00 44.80 0.00 32.00 8.00 22283.00 100.00 11:54:51 AM md5 1.20 0.00 38.40 0.00 32.00 8.00 19750.67 100.00 11:54:51 AM md6 0.40 0.00 0.30 0.00 0.75 0.24 1471.50 24.26 11:54:51 AM md7 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 11:54:51 AM md9 0.00 0.00 0.00 0.00 0.00 362.00 0.00 100.00 11:54:51 AM nvme0n1p1 0.20 0.00 0.80 0.00 4.00 0.00 0.00 0.00 11:54:51 AM DEV tps rkB/s wkB/s dkB/s areq-sz aqu-sz await %util 11:54:56 AM loop0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 11:54:56 AM loop1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 11:54:56 AM loop2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 11:54:56 AM loop3 5.00 0.00 56.00 0.00 11.20 0.00 0.20 0.00 11:54:56 AM sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 11:54:56 AM nvme0n1 5.40 0.00 59.30 0.00 10.98 0.14 0.11 14.14 11:54:56 AM sdb 58.80 6160.80 184.00 0.00 107.90 0.36 117.16 35.94 11:54:56 AM sdc 22.80 6106.40 24.00 0.00 268.88 0.07 20.24 7.24 11:54:56 AM sdd 15.40 5916.80 0.00 0.00 384.21 0.03 2.68 2.80 11:54:56 AM sdg 32.00 5823.20 72.00 0.00 184.22 0.09 10.93 8.58 11:54:56 AM sdh 25.40 5961.60 19.20 0.00 235.46 0.06 34.20 5.88 11:54:56 AM sdi 15.20 5916.80 0.00 0.00 389.26 0.03 2.53 2.90 11:54:56 AM sdj 37.80 6079.20 65.60 0.00 162.56 0.13 93.25 12.82 11:54:56 AM sde 16.20 5928.80 0.00 0.00 365.98 0.05 4.85 5.18 11:54:56 AM sdf 58.80 6154.40 184.00 0.00 107.80 0.41 135.38 41.06 11:54:56 AM md1 11.60 32.00 65.90 0.00 8.44 0.00 0.00 0.00 11:54:56 AM md2 1.20 12.00 0.00 0.00 10.00 0.00 0.00 0.00 11:54:56 AM md4 5.00 25.60 12.80 0.00 7.68 0.00 0.00 0.00 11:54:56 AM md5 0.80 0.00 25.60 0.00 32.00 0.00 0.00 0.00 11:54:56 AM md6 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 11:54:56 AM md7 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 11:54:56 AM md9 14.20 0.00 162.40 0.00 11.44 0.00 0.00 0.00 11:54:56 AM md1 2.00 32.00 0.00 0.00 16.00 101.57 1322202.80 100.00 11:54:56 AM md2 1.20 12.00 0.00 0.00 10.00 0.01 12.50 1.50 11:54:56 AM md4 5.60 25.60 32.00 0.00 10.29 8.32 2425.25 100.00 11:54:56 AM md5 1.60 0.00 51.20 0.00 32.00 8.00 14493.62 100.00 11:54:56 AM md6 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 11:54:56 AM md7 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 11:54:56 AM md9 0.00 0.00 0.00 0.00 0.00 346.78 0.00 100.00 11:54:56 AM nvme0n1p1 5.40 0.00 61.80 0.00 11.44 0.00 0.19 0.10 11:54:56 AM DEV tps rkB/s wkB/s dkB/s areq-sz aqu-sz await %util 11:55:01 AM loop0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 11:55:01 AM loop1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 11:55:01 AM loop2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 11:55:01 AM loop3 0.20 0.00 0.80 0.00 4.00 0.00 0.00 0.00 11:55:01 AM sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 11:55:01 AM nvme0n1 6.20 0.00 25.30 0.00 4.08 0.24 0.16 24.18 11:55:01 AM sdb 32.00 4322.40 112.80 0.00 138.60 0.15 77.39 14.82 11:55:01 AM sdc 15.40 4113.60 17.60 0.00 268.26 0.04 28.73 3.76 11:55:01 AM sdd 10.60 4100.80 0.00 0.00 386.87 0.06 7.79 5.98 11:55:01 AM sdg 18.80 4296.00 20.00 0.00 229.57 0.14 68.47 14.30 11:55:01 AM sdh 21.00 4169.60 16.80 0.00 199.35 0.05 16.46 5.14 11:55:01 AM sdi 10.60 4100.80 0.00 0.00 386.87 0.02 2.83 2.30 11:55:01 AM sdj 18.00 4100.80 58.40 0.00 231.07 0.01 1.07 1.34 11:55:01 AM sde 10.60 4100.80 0.00 0.00 386.87 0.02 3.36 2.44 11:55:01 AM sdf 32.40 4164.80 109.60 0.00 131.93 0.42 403.14 42.40 11:55:01 AM md1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 11:55:01 AM md2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 11:55:01 AM md4 6.20 56.00 19.20 0.00 12.13 0.00 0.00 0.00 11:55:01 AM md5 0.40 0.00 12.80 0.00 32.00 0.00 0.00 0.00 11:55:01 AM md6 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 11:55:01 AM md7 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 11:55:01 AM md9 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 11:55:01 AM md1 0.00 0.00 0.00 0.00 0.00 29.00 0.00 100.00 11:55:01 AM md2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 11:55:01 AM md4 6.60 56.00 32.00 0.00 13.33 9.00 1915.21 100.00 11:55:01 AM md5 0.80 0.00 25.60 0.00 32.00 8.00 8086.00 100.00 11:55:01 AM md6 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 11:55:01 AM md7 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 11:55:01 AM md9 0.20 0.00 3.20 0.00 16.00 288.68 0.00 100.00 11:55:01 AM nvme0n1p1 6.20 0.00 31.40 0.00 5.06 0.00 0.13 0.08 11:55:01 AM DEV tps rkB/s wkB/s dkB/s areq-sz aqu-sz await %util 11:55:06 AM loop0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 11:55:06 AM loop1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 11:55:06 AM loop2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 11:55:06 AM loop3 0.20 0.00 0.80 0.00 4.00 0.00 0.00 0.00 11:55:06 AM sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 11:55:06 AM nvme0n1 6.80 0.00 43.10 0.00 6.34 0.08 0.09 7.66 11:55:06 AM sdb 35.60 5771.20 124.80 0.00 165.62 0.25 49.19 25.02 11:55:06 AM sdc 30.40 5803.20 17.60 0.00 191.47 0.16 22.88 16.42 11:55:06 AM sdd 19.00 5752.80 0.00 0.00 302.78 0.06 4.33 6.38 11:55:06 AM sdg 18.80 5699.20 24.00 0.00 304.43 0.03 2.67 2.60 11:55:06 AM sdh 22.20 5741.60 18.40 0.00 259.46 0.08 13.93 8.06 11:55:06 AM sdi 17.80 5738.40 0.00 0.00 322.38 0.05 3.51 5.22 11:55:06 AM sdj 25.60 5734.40 64.00 0.00 226.50 0.04 11.91 4.40 11:55:06 AM sde 14.60 5693.60 0.00 0.00 389.97 0.03 3.36 3.24 11:55:06 AM sdf 34.20 5730.40 124.00 0.00 171.18 0.35 93.43 34.92 11:55:06 AM md1 0.60 0.80 4.10 0.00 8.17 0.00 0.00 0.00 11:55:06 AM md2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 11:55:06 AM md4 3.00 29.60 12.80 0.00 14.13 0.00 0.00 0.00 11:55:06 AM md5 11.60 96.80 19.20 0.00 10.00 0.00 0.00 0.00 11:55:06 AM md6 4.60 59.20 0.00 0.00 12.87 0.00 0.00 0.00 11:55:06 AM md7 3.40 44.80 0.00 0.00 13.18 0.00 0.00 0.00 11:55:06 AM md9 0.40 0.80 3.20 0.00 10.00 0.00 0.00 0.00 11:55:06 AM md1 0.80 0.80 5.00 0.00 7.25 7.58 487712.50 36.32 11:55:06 AM md2 0.20 0.00 0.20 0.00 1.00 0.15 0.00 14.76 11:55:06 AM md4 3.60 29.60 32.00 0.00 17.11 8.55 1527.72 100.00 11:55:06 AM md5 12.00 96.80 32.00 0.00 10.73 9.18 750.15 100.00 11:55:06 AM md6 4.60 59.20 0.00 0.00 12.87 0.04 7.61 3.34 11:55:06 AM md7 3.40 44.80 0.00 0.00 13.18 0.03 8.71 2.90 11:55:06 AM md9 0.20 0.80 0.00 0.00 4.00 288.34 34519.00 100.00 11:55:06 AM nvme0n1p1 7.00 0.00 43.40 0.00 6.20 0.00 0.03 0.02 Average: DEV tps rkB/s wkB/s dkB/s areq-sz aqu-sz await %util Average: loop0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 Average: loop1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 Average: loop2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 Average: loop3 1.16 0.00 11.84 0.00 10.21 0.00 0.17 0.00 Average: sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 Average: nvme0n1 3.72 0.00 25.70 0.00 6.91 0.12 0.12 11.69 Average: sdb 35.24 4954.24 123.52 0.00 144.09 0.22 79.93 22.26 Average: sdc 19.72 4903.68 17.28 0.00 249.54 0.09 26.44 9.34 Average: sdd 14.00 4882.40 0.32 0.00 348.77 0.04 4.56 4.06 Average: sdg 20.92 4891.68 36.32 0.00 235.56 0.06 16.50 5.61 Average: sdh 20.04 4906.40 16.32 0.00 245.64 0.05 19.39 4.68 Average: sdi 13.68 4879.20 0.00 0.00 356.67 0.03 2.94 2.99 Average: sdj 23.48 4911.04 53.28 0.00 211.43 0.04 33.12 4.25 Average: sde 13.08 4872.64 0.00 0.00 372.53 0.03 3.66 3.14 Average: sdf 35.48 4947.36 124.16 0.00 142.94 0.35 176.59 34.30 Average: md1 2.44 6.56 14.00 0.00 8.43 0.00 0.00 0.00 Average: md2 0.24 2.40 0.00 0.00 10.00 0.00 0.00 0.00 Average: md4 3.00 22.24 14.08 0.00 12.11 0.00 0.00 0.00 Average: md5 2.68 19.36 15.36 0.00 12.96 0.00 0.00 0.00 Average: md6 1.00 11.84 0.06 0.00 11.90 0.00 0.00 0.00 Average: md7 0.68 8.96 0.00 0.00 13.18 0.00 0.00 0.00 Average: md9 3.48 0.16 41.60 0.00 12.00 0.00 0.00 0.00 Average: md1 0.56 6.56 1.00 0.00 13.50 89.63 1083777.00 87.27 Average: md2 0.28 2.40 0.04 0.00 8.71 0.03 10.71 3.25 Average: md4 3.44 22.24 28.16 0.00 14.65 8.37 3658.01 100.00 Average: md5 3.12 19.36 29.44 0.00 15.64 8.24 3997.51 100.00 Average: md6 1.04 11.84 0.10 0.00 11.48 0.12 119.92 12.44 Average: md7 0.68 8.96 0.00 0.00 13.18 0.01 8.71 0.58 Average: md9 0.08 0.16 0.64 0.00 10.00 329.58 3027542.00 100.00 Average: nvme0n1p1 3.76 0.00 27.48 0.00 7.31 0.00 0.11 0.04 Quote Link to comment
JorgeB Posted January 2, 2021 Share Posted January 2, 2021 Kind of strange results, don't usually see usage on the MD devices, you have more than 1 at 100%, all SMR disks, but not all SMR disks are the same, server load is also very high, likely due to i/o wait, pause the check and run the diskspeed docker. Quote Link to comment
JorgeB Posted January 2, 2021 Share Posted January 2, 2021 11:55:06 AM md1 0.80 0.80 5.00 0.00 7.25 7.58 487712.50 36.32 11:55:06 AM md2 0.20 0.00 0.20 0.00 1.00 0.15 0.00 14.76 11:55:06 AM md4 3.60 29.60 32.00 0.00 17.11 8.55 1527.72 100.00 11:55:06 AM md5 12.00 96.80 32.00 0.00 10.73 9.18 750.15 100.00 11:55:06 AM md6 4.60 59.20 0.00 0.00 12.87 0.04 7.61 3.34 11:55:06 AM md7 3.40 44.80 0.00 0.00 13.18 0.03 8.71 2.90 11:55:06 AM md9 0.20 0.80 0.00 0.00 4.00 288.34 34519.00 100.00 These suggest some activity going on to the array besides the parity check, as that should no register on the md devices, especially disks 4, 5 and 9. Quote Link to comment
mattyx Posted January 2, 2021 Author Share Posted January 2, 2021 Oddly, after crossing about 5.5% parity check completion, the parity speed has returned to normal levels (currently ~142MB/sec). I am also seeing more normal CPU utilization (~20%, no cores maxed). I'm not sure why, but I'll take it. I'll keep monitoring this for a few more hours, and will mark this solved if the slow parity issue doesn't return. In the meantime, if there are any other issues I should look into, I welcome the pointers. Thanks to Squid and JorgeB for the help!! M Quote Link to comment
JorgeB Posted January 2, 2021 Share Posted January 2, 2021 Almost certainly there was something else, like a docker or plugin, using the array. Quote Link to comment
mattyx Posted January 2, 2021 Author Share Posted January 2, 2021 Got it. Thanks again for the help! Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.