JorgeB Posted July 1, 2020 Share Posted July 1, 2020 Thanks for doing this, just hope the issue happens again, or it will be all for nothing. Quote Link to comment
Juniper Posted July 1, 2020 Author Share Posted July 1, 2020 Just thought about this: I didn't change anything on the array after I took the 10 TB drive out. It still has the parity from last time on it. If the issue does not happen this time it might be bc the parity on the 10 TB drive could still be valid for parity check. If the issue does not happen, I'll redo it, and this time do a pre-clean of the 10 TB drive first. Quote Link to comment
JonathanM Posted July 1, 2020 Share Posted July 1, 2020 1 hour ago, Juniper said: Just thought about this: I didn't change anything on the array after I took the 10 TB drive out. It still has the parity from last time on it. If the issue does not happen this time it might be bc the parity on the 10 TB drive could still be valid for parity check. If the issue does not happen, I'll redo it, and this time do a pre-clean of the 10 TB drive first. Since you only have 8TB data drives, the correct contents of the 10TB after the 8TB boundary IS all zeros. So, you would want to fill the 10TB with random bits, not preclear it. Quote Link to comment
JorgeB Posted July 1, 2020 Share Posted July 1, 2020 10 minutes ago, jonathanm said: Since you only have 8TB data drives, the correct contents of the 10TB after the 8TB boundary IS all zeros. So, you would want to fill the 10TB with random bits, not preclear it. That's one of the strange things about this, if previously the 10TB disk was precleared it would be all zeros, so after the parity swap the new extra parity section would still be correct, i.e., all zeros, even if for some reason Unraid was failing to zero the extra space when it did the parity copy. Quote Link to comment
Juniper Posted July 1, 2020 Author Share Posted July 1, 2020 (edited) Last time I ran a pre-clear on the 10 TB drive before starting the parity swap. The whole drive was 0s when the procedure started. That is the situation I'll try to reestablish if the issue does not occur this time. Edited July 1, 2020 by Juniper Quote Link to comment
Juniper Posted July 3, 2020 Author Share Posted July 3, 2020 Awesome: Parity Swap resulted in the same errors! Did a Parity Swap and then Parity Check, without rebooting. Attached are: Diagnostics before starting Parity Swap Diagnostics after Parity Swap and Parity Check Screenie at the end of the Parity Check with the result message. Same amount of errors as before. Please let me know what needs to be done now. I'll keep the array running. schiethucken-diagnostics-20200701-0923.zip schiethucken-diagnostics-20200703-1230.zip 1 Quote Link to comment
JorgeB Posted July 3, 2020 Share Posted July 3, 2020 Procedure looks to be all correctly done, and honestly I could never imagine how it could be user error since any deviation from the procedure will abort it or not even work, the sync errors start right after the 8TB mark, and curiously they are not on every block, it's every third one, I'm going to try and duplicate this with similar size disks to see if I can find out what's triggering it, if I can't I'll ask Tom to take a look. Quote Link to comment
Juniper Posted July 3, 2020 Author Share Posted July 3, 2020 (edited) Great. I'll keep the disks from the original configuration around and make sure not to copy too much to the array so I can go back to the original config in case more tests are needed. Edited July 3, 2020 by Juniper 1 Quote Link to comment
JorgeB Posted July 10, 2020 Share Posted July 10, 2020 Just an update, I still can't reproduce this issue, I was looking at other threads that reported the same problem before and it happened with various disk/parity size combos, 2 to 4TB, 4 to 8TB, etc, so and since I could never reproduce it with the small SSDs I have on my test server I tried with 4 and 8TB disks that I had available, it's now past the old parity size and no errors, I'm going to let it finish but I would expect that if there were errors they would have already been detected by now: Next I'll reproduce the exact configuration you had on your server and try again, luckily I can get it exactly the same with the disks I have, but I'll need to remove disks from 3 different servers, so it's going to take a few days/weeks. Quote Link to comment
Juniper Posted July 13, 2020 Author Share Posted July 13, 2020 (edited) Sorry to hear your reproduction efforts have not been fruitful so far. Have finished copying everything to the array. My disk 5 has very little data on it and I can take it out to remake the error producing config again. Let me know when you need me to run more tests at some point. Sorry to hear this is so hard to reproduce. Thank you for all your efforts. Edited July 13, 2020 by Juniper Quote Link to comment
JorgeB Posted August 3, 2020 Share Posted August 3, 2020 OK, there's definitely a problem, I managed to reproduce it with an identical array config: My plan is trying to find out in which circumstances this happens, if it's disk size or config related, and then create a bug report, but since I can't reproduce this with small disks this can be a long and tedious process, I expect that @limetechis very busy at the moment with the latest beta and this isn't an urgent issue but if you have any idea what might be causing this with the info below please advise. Quick summary: -over the last couple of years or so multiple users, more than could be explained by just user error or other issues, have been complaining of sync errors after a parity swap, errors start on the extra parity section, i.e., if old parity was 8TB and new parity is 10TB sync errors start when you pass the 8TB mark on the check -I could never reproduce this before on my test server but @Juniperwas willing to repeat the procedure twice so he/she could post the diags and both times it resulted in sync errors, looking at the diags procedure was correctly done as far as I could see. -I was finally able to reproduce this using the exact same array config as the OP, like all other cases the sync errors start immediately past the old parity size: Aug 2 09:14:08 Tower2 kernel: mdcmd (42): check nocorrect Aug 2 09:14:08 Tower2 kernel: md: recovery thread: check P ... Aug 2 09:15:01 Tower2 sSMTP[4234]: Creating SSL connection to host Aug 2 09:15:01 Tower2 sSMTP[4234]: SSL connection using TLS_AES_256_GCM_SHA384 Aug 2 09:15:03 Tower2 sSMTP[4234]: Sent mail for [email protected] (221 2.0.0 closing connection d11sm13298600wrw.77 - gsmtp) uid=0 username=xxx outbytes=708 Aug 2 09:38:35 Tower2 kernel: mlx4_en: eth2: Link Down Aug 2 10:18:41 Tower2 kernel: mlx4_en: eth2: Link Up Aug 2 13:47:06 Tower2 kernel: mlx4_en: eth2: Link Down Aug 3 00:10:09 Tower2 kernel: md: recovery thread: P incorrect, sector=15628053064 Aug 3 00:10:09 Tower2 kernel: md: recovery thread: P incorrect, sector=15628053072 Aug 3 00:10:09 Tower2 kernel: md: recovery thread: P incorrect, sector=15628053080 Aug 3 00:10:09 Tower2 kernel: md: recovery thread: P incorrect, sector=15628053088 Looking at at hex dump of this first block (after adding the 64 sectors from before the partition starts) you can confirm there's data there: root@Tower2:~# dd if=/dev/sdb skip=15628053128 count=8 | hexdump -C 8+0 records in 8+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 4.0566e-05 s, 101 MB/s 00000000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| * 00000200 8a ea 6e 48 2b 58 30 06 0c 2d 62 37 3e 03 d5 e5 |..nH+X0..-b7>...| 00000210 ae 8a 3d e3 69 01 1e 5a cc 06 9e 4e 14 3f 31 63 |..=.i..Z...N.?1c| 00000220 6c a0 75 b6 67 87 6c b8 41 66 9e 2b d0 85 f8 6f |l.u.g.l.Af.+...o| 00000230 64 b3 fc 7b 3f 5e fe 75 1d d3 f0 71 af 9c 4c da |d..{?^.u...q..L.| 00000240 55 c0 60 97 57 1f a2 75 74 08 ff ba ec df 99 83 |U.`.W..ut.......| 00000250 12 23 09 42 b9 94 b2 54 b9 88 3c f8 b9 61 07 de |.#.B...T..<..a..| 00000260 31 ad e1 1d c6 be 7b 49 c8 8f 81 57 f9 f5 47 f0 |1.....{I...W..G.| 00000270 bc de 1f ec 96 c9 48 1b a0 98 4d 0c 7c 63 dc c6 |......H...M.|c..| 00000280 07 d1 cd d0 cb e6 16 78 10 fc 75 a9 f9 c6 63 ec |.......x..u...c.| 00000290 20 5a 3d b2 1b 42 fd 79 2e 37 9e 45 68 37 0c 83 | Z=..B.y.7.Eh7..| 000002a0 fe c6 ef eb b8 5c 73 37 60 8f e0 64 89 c5 b1 f5 |.....\s7`..d....| 000002b0 78 f0 87 54 95 97 e9 dc 6e 52 7d fc e0 ea 14 fc |x..T....nR}.....| 000002c0 ef e1 7c 0c 7d 37 8d 2a 8b ac b5 92 da 38 0b a6 |..|.}7.*.....8..| 000002d0 dc 52 18 12 d5 bb 3e 87 9b 22 c8 5e 90 c6 b8 35 |.R....>..".^...5| 000002e0 7e fa ab 91 7b e0 50 65 22 8d 23 09 5c 17 b1 84 |~...{.Pe".#.\...| 000002f0 6a 48 58 6d 06 58 23 89 c0 96 c1 a3 49 b6 9f 15 |jHXm.X#.....I...| 00000300 be 5c 7e cf 49 b0 9a 02 c2 f6 d4 82 17 0b de 24 |.\~.I..........$| 00000310 02 27 57 bd 5d 13 6c 41 24 1e 98 7e f0 5b 07 2b |.'W.].lA$..~.[.+| 00000320 bd b5 0c 57 a1 57 c5 fd ac 32 14 94 3d a0 ca b9 |...W.W...2..=...| 00000330 40 00 be 8b d6 c9 c8 5f 71 d2 6f 31 de 35 be df |@......_q.o1.5..| 00000340 73 a5 17 82 c1 2b 9f bc fe a6 b2 65 f3 44 24 d6 |s....+.....e.D$.| 00000350 59 8e 56 f6 e7 49 40 58 86 ac 99 6a 9b c3 df 74 |[email protected]| 00000360 8e 94 56 72 0a fe e5 bb 23 f5 df 9c e4 0d a1 31 |..Vr....#......1| 00000370 27 d9 9b c4 bf ad 78 1e 06 3f 7e 17 94 9f 01 a3 |'.....x..?~.....| 00000380 bd 14 96 50 47 3b a6 03 06 95 b8 08 b1 3d 35 ce |...PG;.......=5.| 00000390 28 f2 ce 2a 5d 64 fa be 5d 6b dc 0e 5b 56 5e 72 |(..*]d..]k..[V^r| 000003a0 ab 62 9a cd be d1 58 3b 74 78 70 53 d6 be f8 ec |.b....X;txpS....| 000003b0 d4 26 aa 44 9c f4 e1 7b c4 2c 9e 6b 5e 0c 9e d4 |.&.D...{.,.k^...| 000003c0 f8 b5 8d 96 d5 09 1f ca 70 c6 26 49 90 9d dd 1f |........p.&I....| 000003d0 56 27 9e b1 33 58 87 9f e3 97 57 97 d9 72 c9 4d |V'..3X....W..r.M| 000003e0 95 c8 fd 2f 8d b5 be 46 d5 08 eb 25 d8 46 cd 55 |.../...F...%.F.U| 000003f0 22 75 4b 34 99 f4 a0 95 81 64 7c c7 53 0f ca a7 |"uK4.....d|.S...| 00000400 15 ec 1d 11 91 09 2a 97 a3 c2 d7 5d ce 43 7e 31 |......*....].C~1| 00000410 49 da b8 8c b0 02 d7 6a f4 c9 f6 a5 79 b0 c2 30 |I......j....y..0| 00000420 7e f1 be c3 7c d9 bc 87 8c 4c 04 16 fb 77 75 07 |~...|....L...wu.| 00000430 09 5d a1 35 af ae 0d 36 fb 0e 9f 9e f7 c8 43 9d |.].5...6......C.| 00000440 bd 17 e7 b2 d9 78 b0 2d af f0 29 a4 8d 9f 8e 7e |.....x.-..)....~| 00000450 b8 e7 2a 94 be fb 4f ca 5a 7b 18 6d 53 29 39 40 |..*...O.Z{.mS)9@| 00000460 ea c2 a4 55 48 7a 19 1d 1f 69 26 eb c0 bb ab 2a |...UHz...i&....*| 00000470 65 09 e6 68 a9 69 86 56 5e 9e 0f d7 f3 b2 7e 21 |e..h.i.V^.....~!| 00000480 33 38 04 ab 25 a1 d1 e4 1d fd 01 7e e8 3b b3 c2 |38..%......~.;..| 00000490 2c 8c ca 6d e4 8a 5a 32 24 a4 fa bb a3 b4 c1 88 |,..m..Z2$.......| 000004a0 51 06 22 c7 7f de 50 f7 9a 4d e4 1a 2e d5 8b 75 |Q."...P..M.....u| 000004b0 9f fe f8 1b 2b 67 cb 59 1a d7 93 db 65 d2 1e 1f |....+g.Y....e...| 000004c0 5c 2f 22 b3 07 1e 51 cf 30 ad 49 90 86 d7 a6 53 |\/"...Q.0.I....S| 000004d0 8c 7b 15 e2 fe 31 76 2d e6 f5 96 d7 fb 8c 4a 94 |.{...1v-......J.| 000004e0 04 ba 24 13 ec 03 a3 13 e3 da b8 61 d5 95 59 be |..$........a..Y.| 000004f0 8a 85 da 98 02 16 67 4f ca d6 02 2c 66 39 41 9a |......gO...,f9A.| 00000500 d3 94 f0 c2 68 59 b3 63 83 78 36 02 85 ec d2 aa |....hY.c.x6.....| 00000510 23 c1 11 e5 2e 6a 8f 1b 74 44 f3 ed f1 2a eb 00 |#....j..tD...*..| 00000520 a4 98 b3 4e a4 58 25 35 1b 95 08 51 06 a4 6a 9d |...N.X%5...Q..j.| 00000530 f9 cd 5f 73 35 30 77 1e 99 c8 7c f5 ca 3f 09 3d |.._s50w...|..?.=| 00000540 94 bc f9 86 e7 af 86 03 66 24 38 ce 5b 2a 10 0c |........f$8.[*..| 00000550 47 9d 68 fc 35 66 b2 70 f2 bd fc 1e db 74 be 25 |G.h.5f.p.....t.%| 00000560 80 ba 4d 11 63 de 34 29 75 ca 9c e5 1d ef 2c ba |..M.c.4)u.....,.| 00000570 3c e2 2f 79 bc 9a 0a e6 d7 b6 22 23 cf c0 2e 89 |<./y......"#....| 00000580 50 93 41 f2 92 e5 da e9 d7 3f 1a b6 dc c6 e6 b6 |P.A......?......| 00000590 98 6e 67 c7 a8 e3 76 43 14 26 02 3b 59 98 be 11 |.ng...vC.&.;Y...| 000005a0 3c 50 2a 6f 36 ea ff f2 6f 7b 53 ee fe 03 cb fb |<P*o6...o{S.....| 000005b0 d1 03 4d 68 5c 25 0c 13 d9 39 6b 14 ed 46 db 41 |..Mh\%...9k..F.A| 000005c0 c8 6e 50 2d 28 88 51 a6 4e 98 f9 df 3e 8c b7 fa |.nP-(.Q.N...>...| 000005d0 2b ca ee 59 1b 80 19 9f 18 e3 a3 b8 c7 d8 00 78 |+..Y...........x| 000005e0 dc cf 64 de d3 2a 9d f1 94 de af 2f 4c 76 82 34 |..d..*...../Lv.4| 000005f0 78 24 0f 91 f8 b2 e5 29 67 1f 88 70 9b cf 1a c4 |x$.....)g..p....| 00000600 f0 8b 8b 23 6b f9 21 df 27 e5 04 6f e9 be a4 fd |...#k.!.'..o....| 00000610 bb 17 54 9c 4a fc cd 3a 0a 54 63 db 0f 97 df eb |..T.J..:.Tc.....| 00000620 84 ae ef 94 2c b2 cd ee 77 ab 13 80 72 02 8b 37 |....,...w...r..7| 00000630 13 e0 04 6d 9c 6f 0a 91 38 6f db d7 75 dc 4d 4f |...m.o..8o..u.MO| 00000640 f4 17 e8 7f 70 b7 32 ab 7b 91 ac ce 2a fe 60 02 |....p.2.{...*.`.| 00000650 77 7b e4 c6 19 36 d1 aa 8c 56 3c b7 b7 91 2b 3b |w{...6...V<...+;| 00000660 de 4a bb 97 74 48 6d 4d b8 4f 69 36 90 f5 2d e8 |.J..tHmM.Oi6..-.| 00000670 35 af 22 e7 73 d8 03 59 1a 09 7b 7b b7 00 86 7e |5.".s..Y..{{...~| 00000680 2e 18 12 c9 a1 3f 9b 4d f7 b3 79 95 c1 96 5d 9b |.....?.M..y...].| 00000690 b2 cf 4e e4 14 40 12 5f 22 75 b6 ed 31 d2 1b 6a |..N..@._"u..1..j| 000006a0 4e f4 d7 35 17 3d ff 95 d2 5e d7 62 53 b5 28 17 |N..5.=...^.bS.(.| 000006b0 b7 f7 64 74 be b6 34 9e b6 92 c4 eb bf 6e 51 02 |..dt..4......nQ.| 000006c0 98 3d 5f e8 a9 d3 87 da 31 83 1b d6 db 35 b8 de |.=_.....1....5..| 000006d0 82 15 9e 14 27 8a 61 95 6b 6d df 6f da 93 2c 06 |....'.a.km.o..,.| 000006e0 63 59 0e 35 3d 68 c1 c9 e1 b1 1b c9 42 cd 4a fa |cY.5=h......B.J.| 000006f0 10 c3 e2 68 1c b6 06 3d db ea 9b 91 48 3b 3c 72 |...h...=....H;<r| 00000700 23 e6 c4 6c 30 44 9c 90 48 69 09 c1 ab 4a 91 ac |#..l0D..Hi...J..| 00000710 20 e2 0b 13 eb 67 56 11 59 99 c5 b0 e9 35 2a dd | ....gV.Y....5*.| 00000720 4c 02 60 5f 3e ec 35 41 ba fe 44 20 b2 5f d6 9a |L.`_>.5A..D ._..| 00000730 13 02 fa 9d ee 9d ec 8a 77 e2 58 17 43 85 51 a6 |........w.X.C.Q.| 00000740 be 0a 3b 10 f1 9f b2 8b 75 d5 b1 7c 7f 04 04 c5 |..;.....u..|....| 00000750 26 ee ea 21 38 9c 0e 30 e4 a9 fd 63 a4 fd bb dd |&..!8..0...c....| 00000760 d3 24 a8 a4 68 ce 4f a5 b2 ec d0 79 38 6f aa b3 |.$..h.O....y8o..| 00000770 6a 87 97 47 aa 57 28 15 06 fe f7 d8 5d c8 fd 5f |j..G.W(.....].._| 00000780 15 bf a9 52 0d 25 ed aa 48 5f 88 28 1d a0 ee fd |...R.%..H_.(....| 00000790 9f 55 5d 61 39 91 45 23 50 b5 98 b0 1e cf c7 df |.U]a9.E#P.......| 000007a0 bc f6 5f b8 07 da 9c 68 81 0b f3 d5 7d 56 84 ce |.._....h....}V..| 000007b0 0f ff ec 7e a8 1f 5a 5b 3c 68 33 c2 e8 fb 68 f1 |...~..Z[<h3...h.| 000007c0 a0 ec 17 51 0f 27 f8 2c c4 b0 a4 8b 88 da f4 34 |...Q.'.,.......4| 000007d0 bf 63 cc 69 38 6c 81 75 5a bf 2b 93 ec 0c f9 e7 |.c.i8l.uZ.+.....| 000007e0 9e dc bd 71 60 5b d9 1f fa dc 0d 23 ca 9a 7e 3f |...q`[.....#..~?| 000007f0 b4 81 21 07 a1 81 de 98 c5 aa 01 42 ba 5a b0 7f |..!........B.Z..| 00000800 44 44 a1 ce e0 b0 e4 73 d5 be 6d 10 a3 77 0f 26 |DD.....s..m..w.&| 00000810 bb cd 5f 2c c8 12 5a 70 57 b2 d1 18 6f be c9 fc |.._,..ZpW...o...| 00000820 7a 9a 6d ad ed 5a 68 a9 34 fa 71 fc 24 d6 47 e9 |z.m..Zh.4.q.$.G.| 00000830 72 1a c0 c3 98 26 28 25 e2 7c c6 9f c0 aa 8b e9 |r....&(%.|......| 00000840 ac b4 c8 c8 3c 4f b9 04 ae eb b4 16 23 9e 21 ce |....<O......#.!.| 00000850 57 b7 36 fd ff 7a b4 e6 2c 24 c5 14 63 6d 86 60 |W.6..z..,$..cm.`| 00000860 c3 03 9c 89 89 28 d4 8a 77 d0 bb ac b5 ee 83 62 |.....(..w......b| 00000870 ae f5 6a ae 0b 6f 0f 17 16 44 fe e1 54 12 ad bf |..j..o...D..T...| 00000880 77 d2 75 62 f1 3f 4b ca e6 5e 23 8a 1f dd b1 01 |w.ub.?K..^#.....| 00000890 7a 5b 52 33 25 b9 70 f6 1b 30 40 c4 b9 41 f5 90 |z[R3%[email protected]..| 000008a0 83 2c de 33 b9 35 25 3b 6d 6f 60 ab 44 27 bf f9 |.,.3.5%;mo`.D'..| 000008b0 4c 1b f2 90 96 52 28 d8 29 d0 39 53 7c b3 dd ac |L....R(.).9S|...| 000008c0 29 87 ed 5a 97 1d 5b ee e9 ba 8d eb 63 e3 29 be |)..Z..[.....c.).| 000008d0 93 4c 1e 60 6b ba 0c 76 91 5f d2 20 41 ce 7e 76 |.L.`k..v._. A.~v| 000008e0 38 78 73 96 92 7e 6e a8 2b be d9 20 0b 8d 54 ff |8xs..~n.+.. ..T.| 000008f0 8f 99 e8 1e 90 e9 ac d2 af a8 94 1d d5 b7 61 80 |..............a.| 00000900 b0 bd 9e 0d 82 07 72 55 1a e0 2d de 72 31 0d 76 |......rU..-.r1.v| 00000910 28 0b a2 61 01 a9 e8 45 0d 21 cf bd 27 f2 55 fa |(..a...E.!..'.U.| 00000920 24 ac 47 5a f2 eb 19 f4 b9 1f 0f 63 dd 95 5a 81 |$.GZ.......c..Z.| 00000930 05 9f d8 d0 6c d9 44 21 e0 3c 42 7b a0 72 19 46 |....l.D!.<B{.r.F| 00000940 46 44 6c 3f c5 ae b9 59 7a c3 00 d8 c4 e8 0b 9b |FDl?...Yz.......| 00000950 fd 1f 6c da 31 71 19 8a 38 7a dd b8 08 f9 83 d1 |..l.1q..8z......| 00000960 c6 9a 06 0a 08 f9 46 b9 05 c3 fa d7 0e 7d 4a 0a |......F......}J.| 00000970 ff 4c a1 f4 c0 35 bd 24 f2 28 06 39 db 9b 73 d4 |.L...5.$.(.9..s.| 00000980 7d 9a 2a bd 2b e3 32 30 64 70 62 09 85 19 8d d4 |}.*.+.20dpb.....| 00000990 f5 a1 ad 20 23 65 c5 aa 6c 7e 1d f4 5b 65 57 1b |... #e..l~..[eW.| 000009a0 e2 29 d0 fb 6f 92 50 bd 6b 09 0b 62 c6 04 04 1d |.)..o.P.k..b....| 000009b0 8a f6 3e b6 1c 04 f2 8d 65 ec 72 ab 86 98 44 fc |..>.....e.r...D.| 000009c0 79 96 62 1b 7e fe 70 9e b3 c2 0a 99 ce 78 03 cf |y.b.~.p......x..| 000009d0 91 7a 05 67 c3 94 68 4c fb 4c 62 0f e3 0a a4 cb |.z.g..hL.Lb.....| 000009e0 db 69 32 23 5d a3 6f 0a a1 52 6a 26 52 34 21 92 |.i2#].o..Rj&R4!.| 000009f0 30 a1 5f 1a c8 1f 87 f0 c0 2d 70 64 aa aa 64 4f |0._......-pd..dO| 00000a00 2c 46 c8 98 f6 5c 7c d3 ef 38 c4 7d 9c 91 20 f3 |,F...\|..8.}.. .| 00000a10 dd 8d e0 25 d5 0a 59 c0 8e a6 ec 72 d5 d4 82 81 |...%..Y....r....| 00000a20 94 f7 98 f2 92 91 37 38 35 d6 46 f5 33 26 02 73 |......785.F.3&.s| 00000a30 75 2e b9 35 53 e4 65 4e c5 e1 c6 aa 36 3b 3e 5f |u..5S.eN....6;>_| 00000a40 40 c3 14 b5 99 ea c8 74 22 62 e5 9f 4f 79 ef 04 |@......t"b..Oy..| 00000a50 c2 d9 c8 1e 98 01 a1 90 37 df d5 77 56 c3 16 b6 |........7..wV...| 00000a60 27 77 f4 f9 0d 01 06 96 55 84 4a 05 b0 73 0c 96 |'w......U.J..s..| 00000a70 a3 50 cd f3 72 b8 71 c2 2e d5 75 4d 8b 6c 21 45 |.P..r.q...uM.l!E| 00000a80 b0 05 be 6f bb 17 ec 78 9b 35 66 2f 0a 27 96 b6 |...o...x.5f/.'..| 00000a90 41 ac cc 04 e9 26 9a a8 f7 ee 67 35 10 f3 ff 64 |A....&....g5...d| 00000aa0 37 d1 fe 55 49 a0 6d 15 ae a0 6a 9c 6e dd b9 73 |7..UI.m...j.n..s| 00000ab0 84 1d df d7 24 e6 d7 2b 87 02 da c4 4b 31 fb c6 |....$..+....K1..| 00000ac0 1c fb e4 6b 2d 2c 69 52 0c d2 9f c7 80 f2 fd c4 |...k-,iR........| 00000ad0 9e 7f 42 f9 97 70 ac 52 81 bd 23 bc 34 4b d3 2f |..B..p.R..#.4K./| 00000ae0 5e 26 cf 45 26 10 c9 a0 99 5e 69 41 93 76 c0 85 |^&.E&....^iA.v..| 00000af0 d5 9a 7b c3 06 57 2f ff 8f 06 62 64 ef e9 52 52 |..{..W/...bd..RR| 00000b00 5b eb df 9d ee a3 16 ac e7 ec 03 6c c2 7b 9f cc |[..........l.{..| 00000b10 4e df 29 c5 8d f9 17 24 bd e0 a0 e0 7c 81 a9 84 |N.)....$....|...| 00000b20 2c 4d 21 8c a6 87 22 c8 ce f8 28 53 8d a2 a6 0d |,M!..."...(S....| 00000b30 99 8a 79 e9 04 84 8d 5d 44 69 6d d9 37 52 5e 8d |..y....]Dim.7R^.| 00000b40 66 9d 06 5a 79 2a aa 63 4e 71 83 5f a6 38 03 fc |f..Zy*.cNq._.8..| 00000b50 06 1a f1 1a 6b 8c c3 fa 1b 1f de 1e 1a de cc 12 |....k...........| 00000b60 b0 10 bb ee f1 8b e0 62 10 5f 28 88 57 6c 6f 64 |.......b._(.Wlod| 00000b70 e9 84 68 c5 bb 13 2e 6a 31 6a fb f3 01 50 de 66 |..h....j1j...P.f| 00000b80 c7 1b 49 c1 59 ee d3 46 7d 98 9b 1d 17 6d 21 a7 |..I.Y..F}....m!.| 00000b90 d2 52 ca 96 63 bb aa dd 95 c9 9a ee 32 44 8b 51 |.R..c.......2D.Q| 00000ba0 6a 6f d5 b2 ac 60 8a c5 a2 71 b8 a6 23 97 68 1b |jo...`...q..#.h.| 00000bb0 df b2 1c 85 f5 36 67 fa 31 5c 6b ae 60 24 80 cf |.....6g.1\k.`$..| 00000bc0 13 c0 3d a6 32 67 dc 6c c4 49 12 ef 95 1f 8a 56 |..=.2g.l.I.....V| 00000bd0 a7 67 42 3d ee 27 4a 81 41 33 84 2d a8 f5 02 6f |.gB=.'J.A3.-...o| 00000be0 d0 f9 c3 e2 22 4c c9 c3 47 86 e4 58 c2 eb 8d b8 |...."L..G..X....| 00000bf0 d2 1e 0d 78 cd 0d 27 91 11 43 f4 24 d1 6e 89 02 |...x..'..C.$.n..| 00000c00 08 70 fd 0b 74 71 42 f4 0a c2 74 4e e8 47 6b 8d |.p..tqB...tN.Gk.| 00000c10 e0 63 aa 21 cf a3 2b 26 01 66 e2 ae 5a f0 37 29 |.c.!..+&.f..Z.7)| 00000c20 e1 a7 ae 0f 23 90 68 92 d4 35 27 64 17 a7 38 f5 |....#.h..5'd..8.| 00000c30 04 17 bb c0 9e 24 3a 46 55 2a 5b 54 d2 76 d1 aa |.....$:FU*[T.v..| 00000c40 d2 7b b4 89 98 44 b4 17 7e e2 23 c8 26 44 59 d4 |.{...D..~.#.&DY.| 00000c50 77 91 7d 7a df f3 3c e0 7e f4 54 72 ce 20 52 dc |w.}z..<.~.Tr. R.| 00000c60 73 c5 31 e2 fc 25 52 01 6f a4 6a 4c 18 0c 24 bc |s.1..%R.o.jL..$.| 00000c70 dd e9 50 fb 1b b1 8a 2d 9b d6 21 b0 3b 95 85 b6 |..P....-..!.;...| 00000c80 25 45 d7 9a 75 4e 28 8f fa dd dc 91 60 07 f7 f8 |%E..uN(.....`...| 00000c90 3c e3 45 5b af 3c b0 b3 c5 8f 27 12 d4 ae 67 c0 |<.E[.<....'...g.| 00000ca0 fd 03 89 41 34 ee 8d 7e 45 b8 14 35 c5 90 d2 02 |...A4..~E..5....| 00000cb0 f5 2b 09 85 37 3e 75 5c 0b 93 a3 d9 72 bd 04 c6 |.+..7>u\....r...| 00000cc0 f1 74 3a 4b ad 6c fc cc 18 3f 83 4d 69 7f ec 55 |.t:K.l...?.Mi..U| 00000cd0 7f db c3 45 64 e9 69 b1 cb 9a a5 ab 08 da 95 56 |...Ed.i........V| 00000ce0 ef 54 3a 1d 60 68 b0 98 e7 f7 80 54 d0 c3 71 7d |.T:.`h.....T..q}| 00000cf0 9c 9e 73 60 46 fb 16 c6 0a 54 31 3f 10 16 fe 1f |..s`F....T1?....| 00000d00 56 f6 b7 79 ac 1b 8a 84 e8 a5 47 2e 5e 5c a0 2d |V..y......G.^\.-| 00000d10 13 f8 72 22 73 9a 98 67 07 66 ff 07 16 0a cb cf |..r"s..g.f......| 00000d20 85 69 16 fb f9 3a 32 a9 3e 89 56 c6 41 b8 1f 66 |.i...:2.>.V.A..f| 00000d30 b4 d1 69 02 15 fa f4 84 08 97 cc d1 14 03 20 52 |..i........... R| 00000d40 12 13 55 bf 02 ac 50 3f c3 65 4f bb cb 2c 1f a8 |..U...P?.eO..,..| 00000d50 a5 25 8e 72 c7 92 d9 93 1e 41 14 68 89 54 bb fb |.%.r.....A.h.T..| 00000d60 c1 44 fd 57 f8 7e 19 4e 37 b4 08 e0 cc b2 7a 89 |.D.W.~.N7.....z.| 00000d70 74 69 2e 91 8a fd af 11 ee 09 07 43 3d 0d 06 8f |ti.........C=...| 00000d80 46 c5 79 e3 d4 69 7e 2e fb c5 1d 77 9b 22 81 c6 |F.y..i~....w."..| 00000d90 c4 df 61 32 1b 95 b3 87 68 1a f6 72 c2 cf 0a 2b |..a2....h..r...+| 00000da0 14 28 15 ac 9e db ab 69 ba f9 8d 1a df 12 95 32 |.(.....i.......2| 00000db0 bb 77 a1 76 cc 8d 1e 17 5c d7 55 d4 84 ef b3 53 |.w.v....\.U....S| 00000dc0 69 79 95 1d 93 f3 a4 df 09 b2 d4 da 16 35 b3 25 |iy...........5.%| 00000dd0 23 70 cd 56 97 28 ce 7d 91 3e 96 da 65 ba ed 3f |#p.V.(.}.>..e..?| 00000de0 a8 61 b2 2a d4 81 11 74 e4 74 53 de ff c6 09 56 |.a.*...t.tS....V| 00000df0 ec a3 d9 04 80 bf ea 92 85 dd 54 8c 31 e4 13 36 |..........T.1..6| 00000e00 5b 9c 79 6f bc 2f f7 a1 80 f8 8d 88 98 08 c3 56 |[.yo./.........V| 00000e10 c8 fb e2 90 14 91 86 57 b9 0b b9 7d 08 3b c7 80 |.......W...}.;..| 00000e20 80 f4 3b 9b cf 93 19 cc b4 4f 06 46 20 26 c8 ea |..;......O.F &..| 00000e30 4d f8 dd bb 6c 87 2b 00 a1 71 96 42 b4 e6 98 3d |M...l.+..q.B...=| 00000e40 4a dc 72 e3 fa ce a4 ba d2 27 cf f5 5d 62 0b 04 |J.r......'..]b..| 00000e50 3f d2 87 6d 53 48 0b d5 3b 1b 4a 9c e7 29 84 fc |?..mSH..;.J..)..| 00000e60 d6 3b 41 b5 65 cf 12 45 14 4f 4d db 4e 8a 61 41 |.;A.e..E.OM.N.aA| 00000e70 e9 11 fe 81 29 78 fa df 99 32 f7 e2 56 68 a4 24 |....)x...2..Vh.$| 00000e80 b1 9a 87 b1 5c 27 8c f4 ae a4 ae 13 67 df c6 27 |....\'......g..'| 00000e90 f2 74 37 a7 85 b1 6e 1b 34 3b 65 b6 a3 8e 8e d4 |.t7...n.4;e.....| 00000ea0 62 46 25 a9 0d da 78 a5 5a 89 03 f5 e1 20 26 b9 |bF%...x.Z.... &.| 00000eb0 c8 44 08 e8 61 be fc b8 cb 78 6d 98 6c 39 ac 7c |.D..a....xm.l9.|| 00000ec0 76 e4 1f 7c 88 c9 04 11 c8 b7 7e e1 f1 c8 ba 2d |v..|......~....-| 00000ed0 a3 0b 6e aa 32 71 a5 c5 d6 02 35 ed b7 c5 b1 62 |..n.2q....5....b| 00000ee0 62 84 c0 ac 0b 40 1d 57 fd 84 53 2b d5 81 34 fb |[email protected]+..4.| 00000ef0 ff c4 87 c8 a4 ac a1 96 76 d1 e9 42 cf 1f cb 42 |........v..B...B| 00000f00 c5 9f 82 e2 35 87 e3 9e ec 2f b7 93 83 cf b6 37 |....5..../.....7| 00000f10 5f d6 ea e1 3c 9a eb 97 bd fc 44 25 66 94 ae c0 |_...<.....D%f...| 00000f20 60 e5 75 d8 1f 9b 58 7d 95 08 69 77 6c aa 36 21 |`.u...X}..iwl.6!| 00000f30 44 da 80 56 78 6d 7a b8 66 e8 3d cd e1 35 7c bd |D..Vxmz.f.=..5|.| 00000f40 5c f7 fa 34 d8 98 bf fc 0e ca a6 20 ac 3d 96 fb |\..4....... .=..| 00000f50 e9 5c 09 20 1f 17 ea 42 5a b7 a7 c1 56 35 72 42 |.\. ...BZ...V5rB| 00000f60 8d 4c 1b 6c 9a 19 e0 5e 7a 7f f5 65 c3 3f 45 e9 |.L.l...^z..e.?E.| 00000f70 40 a0 ed 03 84 3e bb 0b 85 c8 3a cb 7a 53 b7 60 |@....>....:.zS.`| 00000f80 f8 54 2c 7a d7 45 e8 4e c0 97 26 85 fb 0f f7 ba |.T,z.E.N..&.....| 00000f90 c5 98 f5 51 ee 2a 79 61 f3 79 a9 91 7a 6b b7 3e |...Q.*ya.y..zk.>| 00000fa0 e6 b1 7d 90 34 5b d8 40 43 a5 41 df 14 11 1a 37 |..}.4[[email protected]| 00000fb0 5c 70 4f da 34 91 d3 22 70 c0 32 43 44 d3 6a 07 |\pO.4.."p.2CD.j.| 00000fc0 c9 c4 2b d6 86 3b ff e3 fa c6 60 90 55 a7 03 ba |..+..;....`.U...| 00000fd0 55 da 85 04 ed 24 79 90 fc 25 a0 ef 1a 34 ed 08 |U....$y..%...4..| 00000fe0 3d 23 11 5e c0 d8 5b 6c 0b 18 8d f7 d3 ce 9b 55 |=#.^..[l.......U| 00000ff0 ca 5e 2a f7 1f 34 54 14 9b 10 fd 17 5d ab 12 fb |.^*..4T.....]...| 00001000 Curiously there are some good blocks, notice the 20 block jump here: Aug 3 00:10:09 Tower2 kernel: md: recovery thread: P incorrect, sector=15628053488 Aug 3 00:10:09 Tower2 kernel: md: recovery thread: P incorrect, sector=15628053648 And of course the disk really is zeroed in these: root@Tower2:~# dd if=/dev/sdb skip=15628053560 count=152 | hexdump -C 00000000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| * 00013000 152+0 records in 152+0 records out 77824 bytes (78 kB, 76 KiB) copied, 0.000285448 s, 273 MB/s Also curious for me that for the OP parity is wrong every third block (at least in the logged errors): Jul 2 17:00:03 Schiethucken kernel: md: recovery thread: check P ... Jul 2 17:00:09 Schiethucken emhttpd: cmd: /usr/local/emhttp/plugins/dynamix/scripts/tail_log syslog ### [PREVIOUS LINE REPEATED 1 TIMES] ### Jul 3 03:40:01 Schiethucken root: mover: cache not present, or only cache present Jul 3 06:00:38 Schiethucken kernel: md: recovery thread: P corrected, sector=15628053640 Jul 3 06:00:38 Schiethucken kernel: md: recovery thread: P corrected, sector=15628054664 Jul 3 06:00:38 Schiethucken kernel: md: recovery thread: P corrected, sector=15628055688 Jul 3 06:00:38 Schiethucken kernel: md: recovery thread: P corrected, sector=15628056712 Jul 3 06:00:38 Schiethucken kernel: md: recovery thread: P corrected, sector=15628057736 Jul 3 06:00:38 Schiethucken kernel: md: recovery thread: P corrected, sector=15628058760 The OP started with a precleared parity disk the first time, and a parity corrected one the second time, which means that part of the disk started as all zeros both times, and it looked to me by looking at the logs/graphs that when I did it the new parity disk was completely written till the end after the parity copy, so most likely explanation for this is that the new parity disk isn't being correctly zeroed after the parity copy, but I would assume you're using dd to zero out the extra capacity, so can't imagine how this could happen, any ideas? Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.