[SOLVED] 3.8 mil errors in Parity Check


Recommended Posts

Just thought about this:  I didn't change anything on the array after I took the 10 TB drive out. It still has the parity from last time on it. If the issue does not happen this time it might be bc the parity on the 10 TB drive could still be valid for parity check.

 

If the issue does not happen, I'll redo it, and this time do a pre-clean of the 10 TB drive first.

Link to comment
1 hour ago, Juniper said:

Just thought about this:  I didn't change anything on the array after I took the 10 TB drive out. It still has the parity from last time on it. If the issue does not happen this time it might be bc the parity on the 10 TB drive could still be valid for parity check.

 

If the issue does not happen, I'll redo it, and this time do a pre-clean of the 10 TB drive first.

Since you only have 8TB data drives, the correct contents of the 10TB after the 8TB boundary IS all zeros. So, you would want to fill the 10TB with random bits, not preclear it.

Link to comment
10 minutes ago, jonathanm said:

Since you only have 8TB data drives, the correct contents of the 10TB after the 8TB boundary IS all zeros. So, you would want to fill the 10TB with random bits, not preclear it.

That's one of the strange things about this, if previously the 10TB disk was precleared it would be all zeros, so after the parity swap the new extra parity section would still be correct, i.e., all zeros, even if for some reason Unraid was failing to zero the extra space when it did the parity copy.

Link to comment

Last time I ran a pre-clear on the 10 TB drive before starting the parity swap. The whole drive was 0s when the procedure started. That is the situation I'll try to reestablish if the issue does not occur this time.

Edited by Juniper
Link to comment

Awesome: Parity Swap resulted in the same errors! Did a Parity Swap and then Parity Check, without rebooting. Attached are:

 

Diagnostics before starting Parity Swap

Diagnostics after Parity Swap and Parity Check

Screenie at the end of the Parity Check with the result message.

 

Same amount of errors as before. Please let me know what needs to be done now. I'll keep the array running.

Parity Check Result.png

schiethucken-diagnostics-20200701-0923.zip schiethucken-diagnostics-20200703-1230.zip

  • Thanks 1
Link to comment

Procedure looks to be all correctly done, and honestly I could never imagine how it could be user error since any deviation from the procedure will abort it or not even work, the sync errors start right after the 8TB mark, and curiously they are not on every block, it's every third one, I'm going to try and duplicate this with similar size disks to see if I can find out what's triggering it, if I can't I'll ask Tom to take a look.

 

 

Link to comment

Just an update, I still can't reproduce this issue, I was looking at other threads that reported the same problem before and it happened with various disk/parity size combos, 2 to 4TB, 4 to 8TB, etc, so and since I could never reproduce it with the small SSDs I have on my test server I tried with 4 and 8TB disks that I had available, it's now past the old parity size and no errors, I'm going to let it finish but I would expect that if there were errors they would have already been detected by now:

 

imagem.png.7c43d6ea4bf5ba06d7ab220c561ea832.png

 

Next I'll reproduce the exact configuration you had on your server and try again, luckily I can get it exactly the same with the disks I have, but I'll need to remove disks from 3 different servers, so it's going to take a few days/weeks.

Link to comment

Sorry to hear your reproduction efforts have not been fruitful so far. Have finished copying everything to the array. My disk 5 has very little data on it and I can take it out to remake the error producing config again.

Let me know when you need me to run more tests at some point. Sorry to hear this is so hard to reproduce. Thank you for all your efforts.

Edited by Juniper
Link to comment
  • 3 weeks later...

OK, there's definitely a problem, I managed to reproduce it with an identical array config:

 

image.png.48d12622d66855fbcaf7528407260785.png

 

image.png.6a264d5f20b2d22dba9f78dc4b7223be.png

 

My plan is trying to find out in which circumstances this happens, if it's disk size or config related, and then create a bug report, but since I can't reproduce this with small disks this can be a long and tedious process, I expect that @limetechis very busy at the moment with the latest beta and this isn't an urgent issue but if you have any idea what might be causing this with the info below please advise.

 

Quick summary:

-over the last couple of years or so multiple users, more than could be explained by just user error or other issues, have been complaining of sync errors after a parity swap, errors start on the extra parity section, i.e., if old parity was 8TB and new parity is 10TB sync errors start when you pass the 8TB mark on the check

-I could never reproduce this before on my test server but @Juniperwas willing to repeat the procedure twice so he/she could post the diags and both times it resulted in sync errors, looking at the diags procedure was correctly done as far as I could see.

-I was finally able to reproduce this using the exact same array config as the OP, like all other cases the sync errors start immediately past the old parity size:

 

Aug  2 09:14:08 Tower2 kernel: mdcmd (42): check nocorrect
Aug  2 09:14:08 Tower2 kernel: md: recovery thread: check P ...
Aug  2 09:15:01 Tower2 sSMTP[4234]: Creating SSL connection to host
Aug  2 09:15:01 Tower2 sSMTP[4234]: SSL connection using TLS_AES_256_GCM_SHA384
Aug  2 09:15:03 Tower2 sSMTP[4234]: Sent mail for [email protected] (221 2.0.0 closing connection d11sm13298600wrw.77 - gsmtp) uid=0 username=xxx outbytes=708
Aug  2 09:38:35 Tower2 kernel: mlx4_en: eth2: Link Down
Aug  2 10:18:41 Tower2 kernel: mlx4_en: eth2: Link Up
Aug  2 13:47:06 Tower2 kernel: mlx4_en: eth2: Link Down
Aug  3 00:10:09 Tower2 kernel: md: recovery thread: P incorrect, sector=15628053064
Aug  3 00:10:09 Tower2 kernel: md: recovery thread: P incorrect, sector=15628053072
Aug  3 00:10:09 Tower2 kernel: md: recovery thread: P incorrect, sector=15628053080
Aug  3 00:10:09 Tower2 kernel: md: recovery thread: P incorrect, sector=15628053088

 

Looking at at hex dump of this first block (after adding the 64 sectors from before the partition starts) you can confirm there's data there:

 

root@Tower2:~# dd if=/dev/sdb skip=15628053128 count=8 | hexdump -C
8+0 records in
8+0 records out
4096 bytes (4.1 kB, 4.0 KiB) copied, 4.0566e-05 s, 101 MB/s
00000000  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000200  8a ea 6e 48 2b 58 30 06  0c 2d 62 37 3e 03 d5 e5  |..nH+X0..-b7>...|
00000210  ae 8a 3d e3 69 01 1e 5a  cc 06 9e 4e 14 3f 31 63  |..=.i..Z...N.?1c|
00000220  6c a0 75 b6 67 87 6c b8  41 66 9e 2b d0 85 f8 6f  |l.u.g.l.Af.+...o|
00000230  64 b3 fc 7b 3f 5e fe 75  1d d3 f0 71 af 9c 4c da  |d..{?^.u...q..L.|
00000240  55 c0 60 97 57 1f a2 75  74 08 ff ba ec df 99 83  |U.`.W..ut.......|
00000250  12 23 09 42 b9 94 b2 54  b9 88 3c f8 b9 61 07 de  |.#.B...T..<..a..|
00000260  31 ad e1 1d c6 be 7b 49  c8 8f 81 57 f9 f5 47 f0  |1.....{I...W..G.|
00000270  bc de 1f ec 96 c9 48 1b  a0 98 4d 0c 7c 63 dc c6  |......H...M.|c..|
00000280  07 d1 cd d0 cb e6 16 78  10 fc 75 a9 f9 c6 63 ec  |.......x..u...c.|
00000290  20 5a 3d b2 1b 42 fd 79  2e 37 9e 45 68 37 0c 83  | Z=..B.y.7.Eh7..|
000002a0  fe c6 ef eb b8 5c 73 37  60 8f e0 64 89 c5 b1 f5  |.....\s7`..d....|
000002b0  78 f0 87 54 95 97 e9 dc  6e 52 7d fc e0 ea 14 fc  |x..T....nR}.....|
000002c0  ef e1 7c 0c 7d 37 8d 2a  8b ac b5 92 da 38 0b a6  |..|.}7.*.....8..|
000002d0  dc 52 18 12 d5 bb 3e 87  9b 22 c8 5e 90 c6 b8 35  |.R....>..".^...5|
000002e0  7e fa ab 91 7b e0 50 65  22 8d 23 09 5c 17 b1 84  |~...{.Pe".#.\...|
000002f0  6a 48 58 6d 06 58 23 89  c0 96 c1 a3 49 b6 9f 15  |jHXm.X#.....I...|
00000300  be 5c 7e cf 49 b0 9a 02  c2 f6 d4 82 17 0b de 24  |.\~.I..........$|
00000310  02 27 57 bd 5d 13 6c 41  24 1e 98 7e f0 5b 07 2b  |.'W.].lA$..~.[.+|
00000320  bd b5 0c 57 a1 57 c5 fd  ac 32 14 94 3d a0 ca b9  |...W.W...2..=...|
00000330  40 00 be 8b d6 c9 c8 5f  71 d2 6f 31 de 35 be df  |@......_q.o1.5..|
00000340  73 a5 17 82 c1 2b 9f bc  fe a6 b2 65 f3 44 24 d6  |s....+.....e.D$.|
00000350  59 8e 56 f6 e7 49 40 58  86 ac 99 6a 9b c3 df 74  |[email protected]|
00000360  8e 94 56 72 0a fe e5 bb  23 f5 df 9c e4 0d a1 31  |..Vr....#......1|
00000370  27 d9 9b c4 bf ad 78 1e  06 3f 7e 17 94 9f 01 a3  |'.....x..?~.....|
00000380  bd 14 96 50 47 3b a6 03  06 95 b8 08 b1 3d 35 ce  |...PG;.......=5.|
00000390  28 f2 ce 2a 5d 64 fa be  5d 6b dc 0e 5b 56 5e 72  |(..*]d..]k..[V^r|
000003a0  ab 62 9a cd be d1 58 3b  74 78 70 53 d6 be f8 ec  |.b....X;txpS....|
000003b0  d4 26 aa 44 9c f4 e1 7b  c4 2c 9e 6b 5e 0c 9e d4  |.&.D...{.,.k^...|
000003c0  f8 b5 8d 96 d5 09 1f ca  70 c6 26 49 90 9d dd 1f  |........p.&I....|
000003d0  56 27 9e b1 33 58 87 9f  e3 97 57 97 d9 72 c9 4d  |V'..3X....W..r.M|
000003e0  95 c8 fd 2f 8d b5 be 46  d5 08 eb 25 d8 46 cd 55  |.../...F...%.F.U|
000003f0  22 75 4b 34 99 f4 a0 95  81 64 7c c7 53 0f ca a7  |"uK4.....d|.S...|
00000400  15 ec 1d 11 91 09 2a 97  a3 c2 d7 5d ce 43 7e 31  |......*....].C~1|
00000410  49 da b8 8c b0 02 d7 6a  f4 c9 f6 a5 79 b0 c2 30  |I......j....y..0|
00000420  7e f1 be c3 7c d9 bc 87  8c 4c 04 16 fb 77 75 07  |~...|....L...wu.|
00000430  09 5d a1 35 af ae 0d 36  fb 0e 9f 9e f7 c8 43 9d  |.].5...6......C.|
00000440  bd 17 e7 b2 d9 78 b0 2d  af f0 29 a4 8d 9f 8e 7e  |.....x.-..)....~|
00000450  b8 e7 2a 94 be fb 4f ca  5a 7b 18 6d 53 29 39 40  |..*...O.Z{.mS)9@|
00000460  ea c2 a4 55 48 7a 19 1d  1f 69 26 eb c0 bb ab 2a  |...UHz...i&....*|
00000470  65 09 e6 68 a9 69 86 56  5e 9e 0f d7 f3 b2 7e 21  |e..h.i.V^.....~!|
00000480  33 38 04 ab 25 a1 d1 e4  1d fd 01 7e e8 3b b3 c2  |38..%......~.;..|
00000490  2c 8c ca 6d e4 8a 5a 32  24 a4 fa bb a3 b4 c1 88  |,..m..Z2$.......|
000004a0  51 06 22 c7 7f de 50 f7  9a 4d e4 1a 2e d5 8b 75  |Q."...P..M.....u|
000004b0  9f fe f8 1b 2b 67 cb 59  1a d7 93 db 65 d2 1e 1f  |....+g.Y....e...|
000004c0  5c 2f 22 b3 07 1e 51 cf  30 ad 49 90 86 d7 a6 53  |\/"...Q.0.I....S|
000004d0  8c 7b 15 e2 fe 31 76 2d  e6 f5 96 d7 fb 8c 4a 94  |.{...1v-......J.|
000004e0  04 ba 24 13 ec 03 a3 13  e3 da b8 61 d5 95 59 be  |..$........a..Y.|
000004f0  8a 85 da 98 02 16 67 4f  ca d6 02 2c 66 39 41 9a  |......gO...,f9A.|
00000500  d3 94 f0 c2 68 59 b3 63  83 78 36 02 85 ec d2 aa  |....hY.c.x6.....|
00000510  23 c1 11 e5 2e 6a 8f 1b  74 44 f3 ed f1 2a eb 00  |#....j..tD...*..|
00000520  a4 98 b3 4e a4 58 25 35  1b 95 08 51 06 a4 6a 9d  |...N.X%5...Q..j.|
00000530  f9 cd 5f 73 35 30 77 1e  99 c8 7c f5 ca 3f 09 3d  |.._s50w...|..?.=|
00000540  94 bc f9 86 e7 af 86 03  66 24 38 ce 5b 2a 10 0c  |........f$8.[*..|
00000550  47 9d 68 fc 35 66 b2 70  f2 bd fc 1e db 74 be 25  |G.h.5f.p.....t.%|
00000560  80 ba 4d 11 63 de 34 29  75 ca 9c e5 1d ef 2c ba  |..M.c.4)u.....,.|
00000570  3c e2 2f 79 bc 9a 0a e6  d7 b6 22 23 cf c0 2e 89  |<./y......"#....|
00000580  50 93 41 f2 92 e5 da e9  d7 3f 1a b6 dc c6 e6 b6  |P.A......?......|
00000590  98 6e 67 c7 a8 e3 76 43  14 26 02 3b 59 98 be 11  |.ng...vC.&.;Y...|
000005a0  3c 50 2a 6f 36 ea ff f2  6f 7b 53 ee fe 03 cb fb  |<P*o6...o{S.....|
000005b0  d1 03 4d 68 5c 25 0c 13  d9 39 6b 14 ed 46 db 41  |..Mh\%...9k..F.A|
000005c0  c8 6e 50 2d 28 88 51 a6  4e 98 f9 df 3e 8c b7 fa  |.nP-(.Q.N...>...|
000005d0  2b ca ee 59 1b 80 19 9f  18 e3 a3 b8 c7 d8 00 78  |+..Y...........x|
000005e0  dc cf 64 de d3 2a 9d f1  94 de af 2f 4c 76 82 34  |..d..*...../Lv.4|
000005f0  78 24 0f 91 f8 b2 e5 29  67 1f 88 70 9b cf 1a c4  |x$.....)g..p....|
00000600  f0 8b 8b 23 6b f9 21 df  27 e5 04 6f e9 be a4 fd  |...#k.!.'..o....|
00000610  bb 17 54 9c 4a fc cd 3a  0a 54 63 db 0f 97 df eb  |..T.J..:.Tc.....|
00000620  84 ae ef 94 2c b2 cd ee  77 ab 13 80 72 02 8b 37  |....,...w...r..7|
00000630  13 e0 04 6d 9c 6f 0a 91  38 6f db d7 75 dc 4d 4f  |...m.o..8o..u.MO|
00000640  f4 17 e8 7f 70 b7 32 ab  7b 91 ac ce 2a fe 60 02  |....p.2.{...*.`.|
00000650  77 7b e4 c6 19 36 d1 aa  8c 56 3c b7 b7 91 2b 3b  |w{...6...V<...+;|
00000660  de 4a bb 97 74 48 6d 4d  b8 4f 69 36 90 f5 2d e8  |.J..tHmM.Oi6..-.|
00000670  35 af 22 e7 73 d8 03 59  1a 09 7b 7b b7 00 86 7e  |5.".s..Y..{{...~|
00000680  2e 18 12 c9 a1 3f 9b 4d  f7 b3 79 95 c1 96 5d 9b  |.....?.M..y...].|
00000690  b2 cf 4e e4 14 40 12 5f  22 75 b6 ed 31 d2 1b 6a  |..N..@._"u..1..j|
000006a0  4e f4 d7 35 17 3d ff 95  d2 5e d7 62 53 b5 28 17  |N..5.=...^.bS.(.|
000006b0  b7 f7 64 74 be b6 34 9e  b6 92 c4 eb bf 6e 51 02  |..dt..4......nQ.|
000006c0  98 3d 5f e8 a9 d3 87 da  31 83 1b d6 db 35 b8 de  |.=_.....1....5..|
000006d0  82 15 9e 14 27 8a 61 95  6b 6d df 6f da 93 2c 06  |....'.a.km.o..,.|
000006e0  63 59 0e 35 3d 68 c1 c9  e1 b1 1b c9 42 cd 4a fa  |cY.5=h......B.J.|
000006f0  10 c3 e2 68 1c b6 06 3d  db ea 9b 91 48 3b 3c 72  |...h...=....H;<r|
00000700  23 e6 c4 6c 30 44 9c 90  48 69 09 c1 ab 4a 91 ac  |#..l0D..Hi...J..|
00000710  20 e2 0b 13 eb 67 56 11  59 99 c5 b0 e9 35 2a dd  | ....gV.Y....5*.|
00000720  4c 02 60 5f 3e ec 35 41  ba fe 44 20 b2 5f d6 9a  |L.`_>.5A..D ._..|
00000730  13 02 fa 9d ee 9d ec 8a  77 e2 58 17 43 85 51 a6  |........w.X.C.Q.|
00000740  be 0a 3b 10 f1 9f b2 8b  75 d5 b1 7c 7f 04 04 c5  |..;.....u..|....|
00000750  26 ee ea 21 38 9c 0e 30  e4 a9 fd 63 a4 fd bb dd  |&..!8..0...c....|
00000760  d3 24 a8 a4 68 ce 4f a5  b2 ec d0 79 38 6f aa b3  |.$..h.O....y8o..|
00000770  6a 87 97 47 aa 57 28 15  06 fe f7 d8 5d c8 fd 5f  |j..G.W(.....].._|
00000780  15 bf a9 52 0d 25 ed aa  48 5f 88 28 1d a0 ee fd  |...R.%..H_.(....|
00000790  9f 55 5d 61 39 91 45 23  50 b5 98 b0 1e cf c7 df  |.U]a9.E#P.......|
000007a0  bc f6 5f b8 07 da 9c 68  81 0b f3 d5 7d 56 84 ce  |.._....h....}V..|
000007b0  0f ff ec 7e a8 1f 5a 5b  3c 68 33 c2 e8 fb 68 f1  |...~..Z[<h3...h.|
000007c0  a0 ec 17 51 0f 27 f8 2c  c4 b0 a4 8b 88 da f4 34  |...Q.'.,.......4|
000007d0  bf 63 cc 69 38 6c 81 75  5a bf 2b 93 ec 0c f9 e7  |.c.i8l.uZ.+.....|
000007e0  9e dc bd 71 60 5b d9 1f  fa dc 0d 23 ca 9a 7e 3f  |...q`[.....#..~?|
000007f0  b4 81 21 07 a1 81 de 98  c5 aa 01 42 ba 5a b0 7f  |..!........B.Z..|
00000800  44 44 a1 ce e0 b0 e4 73  d5 be 6d 10 a3 77 0f 26  |DD.....s..m..w.&|
00000810  bb cd 5f 2c c8 12 5a 70  57 b2 d1 18 6f be c9 fc  |.._,..ZpW...o...|
00000820  7a 9a 6d ad ed 5a 68 a9  34 fa 71 fc 24 d6 47 e9  |z.m..Zh.4.q.$.G.|
00000830  72 1a c0 c3 98 26 28 25  e2 7c c6 9f c0 aa 8b e9  |r....&(%.|......|
00000840  ac b4 c8 c8 3c 4f b9 04  ae eb b4 16 23 9e 21 ce  |....<O......#.!.|
00000850  57 b7 36 fd ff 7a b4 e6  2c 24 c5 14 63 6d 86 60  |W.6..z..,$..cm.`|
00000860  c3 03 9c 89 89 28 d4 8a  77 d0 bb ac b5 ee 83 62  |.....(..w......b|
00000870  ae f5 6a ae 0b 6f 0f 17  16 44 fe e1 54 12 ad bf  |..j..o...D..T...|
00000880  77 d2 75 62 f1 3f 4b ca  e6 5e 23 8a 1f dd b1 01  |w.ub.?K..^#.....|
00000890  7a 5b 52 33 25 b9 70 f6  1b 30 40 c4 b9 41 f5 90  |z[R3%[email protected]..|
000008a0  83 2c de 33 b9 35 25 3b  6d 6f 60 ab 44 27 bf f9  |.,.3.5%;mo`.D'..|
000008b0  4c 1b f2 90 96 52 28 d8  29 d0 39 53 7c b3 dd ac  |L....R(.).9S|...|
000008c0  29 87 ed 5a 97 1d 5b ee  e9 ba 8d eb 63 e3 29 be  |)..Z..[.....c.).|
000008d0  93 4c 1e 60 6b ba 0c 76  91 5f d2 20 41 ce 7e 76  |.L.`k..v._. A.~v|
000008e0  38 78 73 96 92 7e 6e a8  2b be d9 20 0b 8d 54 ff  |8xs..~n.+.. ..T.|
000008f0  8f 99 e8 1e 90 e9 ac d2  af a8 94 1d d5 b7 61 80  |..............a.|
00000900  b0 bd 9e 0d 82 07 72 55  1a e0 2d de 72 31 0d 76  |......rU..-.r1.v|
00000910  28 0b a2 61 01 a9 e8 45  0d 21 cf bd 27 f2 55 fa  |(..a...E.!..'.U.|
00000920  24 ac 47 5a f2 eb 19 f4  b9 1f 0f 63 dd 95 5a 81  |$.GZ.......c..Z.|
00000930  05 9f d8 d0 6c d9 44 21  e0 3c 42 7b a0 72 19 46  |....l.D!.<B{.r.F|
00000940  46 44 6c 3f c5 ae b9 59  7a c3 00 d8 c4 e8 0b 9b  |FDl?...Yz.......|
00000950  fd 1f 6c da 31 71 19 8a  38 7a dd b8 08 f9 83 d1  |..l.1q..8z......|
00000960  c6 9a 06 0a 08 f9 46 b9  05 c3 fa d7 0e 7d 4a 0a  |......F......}J.|
00000970  ff 4c a1 f4 c0 35 bd 24  f2 28 06 39 db 9b 73 d4  |.L...5.$.(.9..s.|
00000980  7d 9a 2a bd 2b e3 32 30  64 70 62 09 85 19 8d d4  |}.*.+.20dpb.....|
00000990  f5 a1 ad 20 23 65 c5 aa  6c 7e 1d f4 5b 65 57 1b  |... #e..l~..[eW.|
000009a0  e2 29 d0 fb 6f 92 50 bd  6b 09 0b 62 c6 04 04 1d  |.)..o.P.k..b....|
000009b0  8a f6 3e b6 1c 04 f2 8d  65 ec 72 ab 86 98 44 fc  |..>.....e.r...D.|
000009c0  79 96 62 1b 7e fe 70 9e  b3 c2 0a 99 ce 78 03 cf  |y.b.~.p......x..|
000009d0  91 7a 05 67 c3 94 68 4c  fb 4c 62 0f e3 0a a4 cb  |.z.g..hL.Lb.....|
000009e0  db 69 32 23 5d a3 6f 0a  a1 52 6a 26 52 34 21 92  |.i2#].o..Rj&R4!.|
000009f0  30 a1 5f 1a c8 1f 87 f0  c0 2d 70 64 aa aa 64 4f  |0._......-pd..dO|
00000a00  2c 46 c8 98 f6 5c 7c d3  ef 38 c4 7d 9c 91 20 f3  |,F...\|..8.}.. .|
00000a10  dd 8d e0 25 d5 0a 59 c0  8e a6 ec 72 d5 d4 82 81  |...%..Y....r....|
00000a20  94 f7 98 f2 92 91 37 38  35 d6 46 f5 33 26 02 73  |......785.F.3&.s|
00000a30  75 2e b9 35 53 e4 65 4e  c5 e1 c6 aa 36 3b 3e 5f  |u..5S.eN....6;>_|
00000a40  40 c3 14 b5 99 ea c8 74  22 62 e5 9f 4f 79 ef 04  |@......t"b..Oy..|
00000a50  c2 d9 c8 1e 98 01 a1 90  37 df d5 77 56 c3 16 b6  |........7..wV...|
00000a60  27 77 f4 f9 0d 01 06 96  55 84 4a 05 b0 73 0c 96  |'w......U.J..s..|
00000a70  a3 50 cd f3 72 b8 71 c2  2e d5 75 4d 8b 6c 21 45  |.P..r.q...uM.l!E|
00000a80  b0 05 be 6f bb 17 ec 78  9b 35 66 2f 0a 27 96 b6  |...o...x.5f/.'..|
00000a90  41 ac cc 04 e9 26 9a a8  f7 ee 67 35 10 f3 ff 64  |A....&....g5...d|
00000aa0  37 d1 fe 55 49 a0 6d 15  ae a0 6a 9c 6e dd b9 73  |7..UI.m...j.n..s|
00000ab0  84 1d df d7 24 e6 d7 2b  87 02 da c4 4b 31 fb c6  |....$..+....K1..|
00000ac0  1c fb e4 6b 2d 2c 69 52  0c d2 9f c7 80 f2 fd c4  |...k-,iR........|
00000ad0  9e 7f 42 f9 97 70 ac 52  81 bd 23 bc 34 4b d3 2f  |..B..p.R..#.4K./|
00000ae0  5e 26 cf 45 26 10 c9 a0  99 5e 69 41 93 76 c0 85  |^&.E&....^iA.v..|
00000af0  d5 9a 7b c3 06 57 2f ff  8f 06 62 64 ef e9 52 52  |..{..W/...bd..RR|
00000b00  5b eb df 9d ee a3 16 ac  e7 ec 03 6c c2 7b 9f cc  |[..........l.{..|
00000b10  4e df 29 c5 8d f9 17 24  bd e0 a0 e0 7c 81 a9 84  |N.)....$....|...|
00000b20  2c 4d 21 8c a6 87 22 c8  ce f8 28 53 8d a2 a6 0d  |,M!..."...(S....|
00000b30  99 8a 79 e9 04 84 8d 5d  44 69 6d d9 37 52 5e 8d  |..y....]Dim.7R^.|
00000b40  66 9d 06 5a 79 2a aa 63  4e 71 83 5f a6 38 03 fc  |f..Zy*.cNq._.8..|
00000b50  06 1a f1 1a 6b 8c c3 fa  1b 1f de 1e 1a de cc 12  |....k...........|
00000b60  b0 10 bb ee f1 8b e0 62  10 5f 28 88 57 6c 6f 64  |.......b._(.Wlod|
00000b70  e9 84 68 c5 bb 13 2e 6a  31 6a fb f3 01 50 de 66  |..h....j1j...P.f|
00000b80  c7 1b 49 c1 59 ee d3 46  7d 98 9b 1d 17 6d 21 a7  |..I.Y..F}....m!.|
00000b90  d2 52 ca 96 63 bb aa dd  95 c9 9a ee 32 44 8b 51  |.R..c.......2D.Q|
00000ba0  6a 6f d5 b2 ac 60 8a c5  a2 71 b8 a6 23 97 68 1b  |jo...`...q..#.h.|
00000bb0  df b2 1c 85 f5 36 67 fa  31 5c 6b ae 60 24 80 cf  |.....6g.1\k.`$..|
00000bc0  13 c0 3d a6 32 67 dc 6c  c4 49 12 ef 95 1f 8a 56  |..=.2g.l.I.....V|
00000bd0  a7 67 42 3d ee 27 4a 81  41 33 84 2d a8 f5 02 6f  |.gB=.'J.A3.-...o|
00000be0  d0 f9 c3 e2 22 4c c9 c3  47 86 e4 58 c2 eb 8d b8  |...."L..G..X....|
00000bf0  d2 1e 0d 78 cd 0d 27 91  11 43 f4 24 d1 6e 89 02  |...x..'..C.$.n..|
00000c00  08 70 fd 0b 74 71 42 f4  0a c2 74 4e e8 47 6b 8d  |.p..tqB...tN.Gk.|
00000c10  e0 63 aa 21 cf a3 2b 26  01 66 e2 ae 5a f0 37 29  |.c.!..+&.f..Z.7)|
00000c20  e1 a7 ae 0f 23 90 68 92  d4 35 27 64 17 a7 38 f5  |....#.h..5'd..8.|
00000c30  04 17 bb c0 9e 24 3a 46  55 2a 5b 54 d2 76 d1 aa  |.....$:FU*[T.v..|
00000c40  d2 7b b4 89 98 44 b4 17  7e e2 23 c8 26 44 59 d4  |.{...D..~.#.&DY.|
00000c50  77 91 7d 7a df f3 3c e0  7e f4 54 72 ce 20 52 dc  |w.}z..<.~.Tr. R.|
00000c60  73 c5 31 e2 fc 25 52 01  6f a4 6a 4c 18 0c 24 bc  |s.1..%R.o.jL..$.|
00000c70  dd e9 50 fb 1b b1 8a 2d  9b d6 21 b0 3b 95 85 b6  |..P....-..!.;...|
00000c80  25 45 d7 9a 75 4e 28 8f  fa dd dc 91 60 07 f7 f8  |%E..uN(.....`...|
00000c90  3c e3 45 5b af 3c b0 b3  c5 8f 27 12 d4 ae 67 c0  |<.E[.<....'...g.|
00000ca0  fd 03 89 41 34 ee 8d 7e  45 b8 14 35 c5 90 d2 02  |...A4..~E..5....|
00000cb0  f5 2b 09 85 37 3e 75 5c  0b 93 a3 d9 72 bd 04 c6  |.+..7>u\....r...|
00000cc0  f1 74 3a 4b ad 6c fc cc  18 3f 83 4d 69 7f ec 55  |.t:K.l...?.Mi..U|
00000cd0  7f db c3 45 64 e9 69 b1  cb 9a a5 ab 08 da 95 56  |...Ed.i........V|
00000ce0  ef 54 3a 1d 60 68 b0 98  e7 f7 80 54 d0 c3 71 7d  |.T:.`h.....T..q}|
00000cf0  9c 9e 73 60 46 fb 16 c6  0a 54 31 3f 10 16 fe 1f  |..s`F....T1?....|
00000d00  56 f6 b7 79 ac 1b 8a 84  e8 a5 47 2e 5e 5c a0 2d  |V..y......G.^\.-|
00000d10  13 f8 72 22 73 9a 98 67  07 66 ff 07 16 0a cb cf  |..r"s..g.f......|
00000d20  85 69 16 fb f9 3a 32 a9  3e 89 56 c6 41 b8 1f 66  |.i...:2.>.V.A..f|
00000d30  b4 d1 69 02 15 fa f4 84  08 97 cc d1 14 03 20 52  |..i........... R|
00000d40  12 13 55 bf 02 ac 50 3f  c3 65 4f bb cb 2c 1f a8  |..U...P?.eO..,..|
00000d50  a5 25 8e 72 c7 92 d9 93  1e 41 14 68 89 54 bb fb  |.%.r.....A.h.T..|
00000d60  c1 44 fd 57 f8 7e 19 4e  37 b4 08 e0 cc b2 7a 89  |.D.W.~.N7.....z.|
00000d70  74 69 2e 91 8a fd af 11  ee 09 07 43 3d 0d 06 8f  |ti.........C=...|
00000d80  46 c5 79 e3 d4 69 7e 2e  fb c5 1d 77 9b 22 81 c6  |F.y..i~....w."..|
00000d90  c4 df 61 32 1b 95 b3 87  68 1a f6 72 c2 cf 0a 2b  |..a2....h..r...+|
00000da0  14 28 15 ac 9e db ab 69  ba f9 8d 1a df 12 95 32  |.(.....i.......2|
00000db0  bb 77 a1 76 cc 8d 1e 17  5c d7 55 d4 84 ef b3 53  |.w.v....\.U....S|
00000dc0  69 79 95 1d 93 f3 a4 df  09 b2 d4 da 16 35 b3 25  |iy...........5.%|
00000dd0  23 70 cd 56 97 28 ce 7d  91 3e 96 da 65 ba ed 3f  |#p.V.(.}.>..e..?|
00000de0  a8 61 b2 2a d4 81 11 74  e4 74 53 de ff c6 09 56  |.a.*...t.tS....V|
00000df0  ec a3 d9 04 80 bf ea 92  85 dd 54 8c 31 e4 13 36  |..........T.1..6|
00000e00  5b 9c 79 6f bc 2f f7 a1  80 f8 8d 88 98 08 c3 56  |[.yo./.........V|
00000e10  c8 fb e2 90 14 91 86 57  b9 0b b9 7d 08 3b c7 80  |.......W...}.;..|
00000e20  80 f4 3b 9b cf 93 19 cc  b4 4f 06 46 20 26 c8 ea  |..;......O.F &..|
00000e30  4d f8 dd bb 6c 87 2b 00  a1 71 96 42 b4 e6 98 3d  |M...l.+..q.B...=|
00000e40  4a dc 72 e3 fa ce a4 ba  d2 27 cf f5 5d 62 0b 04  |J.r......'..]b..|
00000e50  3f d2 87 6d 53 48 0b d5  3b 1b 4a 9c e7 29 84 fc  |?..mSH..;.J..)..|
00000e60  d6 3b 41 b5 65 cf 12 45  14 4f 4d db 4e 8a 61 41  |.;A.e..E.OM.N.aA|
00000e70  e9 11 fe 81 29 78 fa df  99 32 f7 e2 56 68 a4 24  |....)x...2..Vh.$|
00000e80  b1 9a 87 b1 5c 27 8c f4  ae a4 ae 13 67 df c6 27  |....\'......g..'|
00000e90  f2 74 37 a7 85 b1 6e 1b  34 3b 65 b6 a3 8e 8e d4  |.t7...n.4;e.....|
00000ea0  62 46 25 a9 0d da 78 a5  5a 89 03 f5 e1 20 26 b9  |bF%...x.Z.... &.|
00000eb0  c8 44 08 e8 61 be fc b8  cb 78 6d 98 6c 39 ac 7c  |.D..a....xm.l9.||
00000ec0  76 e4 1f 7c 88 c9 04 11  c8 b7 7e e1 f1 c8 ba 2d  |v..|......~....-|
00000ed0  a3 0b 6e aa 32 71 a5 c5  d6 02 35 ed b7 c5 b1 62  |..n.2q....5....b|
00000ee0  62 84 c0 ac 0b 40 1d 57  fd 84 53 2b d5 81 34 fb  |[email protected]+..4.|
00000ef0  ff c4 87 c8 a4 ac a1 96  76 d1 e9 42 cf 1f cb 42  |........v..B...B|
00000f00  c5 9f 82 e2 35 87 e3 9e  ec 2f b7 93 83 cf b6 37  |....5..../.....7|
00000f10  5f d6 ea e1 3c 9a eb 97  bd fc 44 25 66 94 ae c0  |_...<.....D%f...|
00000f20  60 e5 75 d8 1f 9b 58 7d  95 08 69 77 6c aa 36 21  |`.u...X}..iwl.6!|
00000f30  44 da 80 56 78 6d 7a b8  66 e8 3d cd e1 35 7c bd  |D..Vxmz.f.=..5|.|
00000f40  5c f7 fa 34 d8 98 bf fc  0e ca a6 20 ac 3d 96 fb  |\..4....... .=..|
00000f50  e9 5c 09 20 1f 17 ea 42  5a b7 a7 c1 56 35 72 42  |.\. ...BZ...V5rB|
00000f60  8d 4c 1b 6c 9a 19 e0 5e  7a 7f f5 65 c3 3f 45 e9  |.L.l...^z..e.?E.|
00000f70  40 a0 ed 03 84 3e bb 0b  85 c8 3a cb 7a 53 b7 60  |@....>....:.zS.`|
00000f80  f8 54 2c 7a d7 45 e8 4e  c0 97 26 85 fb 0f f7 ba  |.T,z.E.N..&.....|
00000f90  c5 98 f5 51 ee 2a 79 61  f3 79 a9 91 7a 6b b7 3e  |...Q.*ya.y..zk.>|
00000fa0  e6 b1 7d 90 34 5b d8 40  43 a5 41 df 14 11 1a 37  |..}.4[[email protected]|
00000fb0  5c 70 4f da 34 91 d3 22  70 c0 32 43 44 d3 6a 07  |\pO.4.."p.2CD.j.|
00000fc0  c9 c4 2b d6 86 3b ff e3  fa c6 60 90 55 a7 03 ba  |..+..;....`.U...|
00000fd0  55 da 85 04 ed 24 79 90  fc 25 a0 ef 1a 34 ed 08  |U....$y..%...4..|
00000fe0  3d 23 11 5e c0 d8 5b 6c  0b 18 8d f7 d3 ce 9b 55  |=#.^..[l.......U|
00000ff0  ca 5e 2a f7 1f 34 54 14  9b 10 fd 17 5d ab 12 fb  |.^*..4T.....]...|
00001000

 

Curiously there are some good blocks, notice the 20 block jump here:

Aug  3 00:10:09 Tower2 kernel: md: recovery thread: P incorrect, sector=15628053488
Aug  3 00:10:09 Tower2 kernel: md: recovery thread: P incorrect, sector=15628053648

 

And of course the disk really is zeroed in these:

root@Tower2:~# dd if=/dev/sdb skip=15628053560 count=152 | hexdump -C
00000000  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00013000
152+0 records in
152+0 records out
77824 bytes (78 kB, 76 KiB) copied, 0.000285448 s, 273 MB/s

 

Also curious for me that for the OP parity is wrong every third block (at least in the logged errors):

Jul  2 17:00:03 Schiethucken kernel: md: recovery thread: check P ...
Jul  2 17:00:09 Schiethucken emhttpd: cmd: /usr/local/emhttp/plugins/dynamix/scripts/tail_log syslog
### [PREVIOUS LINE REPEATED 1 TIMES] ###
Jul  3 03:40:01 Schiethucken root: mover: cache not present, or only cache present
Jul  3 06:00:38 Schiethucken kernel: md: recovery thread: P corrected, sector=15628053640
Jul  3 06:00:38 Schiethucken kernel: md: recovery thread: P corrected, sector=15628054664
Jul  3 06:00:38 Schiethucken kernel: md: recovery thread: P corrected, sector=15628055688
Jul  3 06:00:38 Schiethucken kernel: md: recovery thread: P corrected, sector=15628056712
Jul  3 06:00:38 Schiethucken kernel: md: recovery thread: P corrected, sector=15628057736
Jul  3 06:00:38 Schiethucken kernel: md: recovery thread: P corrected, sector=15628058760

 

 

The OP started with a precleared parity disk the first time, and a parity corrected one the second time, which means that part of the disk started as all zeros both times, and it looked to me by looking at the logs/graphs that when I did it the new parity disk was completely written till the end after the parity copy, so most likely explanation for this is that the new parity disk isn't being correctly zeroed after the parity copy, but I would assume you're using dd to zero out the extra capacity, so can't imagine how this could happen, any ideas?

 

 

 

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.