June 21, 201115 yr Hey guys, I did a small upgrade (1.5tb to 2.0tb drive) on my unraid box, specs as follows: SuperMicro X7SPA-HF 2x 2gig Kingston sodimms. sda - WD20EARS sdb - WD20EARS sdc - WD15EARS originally, now upgraded to WD20EARS sde - WD15EARS Memtest was run overnight before the upgrade and passed fine. To be extra careful, I ran a reiserfsck on each data drive (not the parity) one drive at a time, and this passed with no errors. A partiy check was then run, and passed fine with no sync errors. I followed the procedure in the Wiki to upgrade a drive - power down, remove old drive, insert new drive, power up, start array and let it rebuild & expand. This occured with no errors. Once this was done, I ran another parity check as I'd like to replace sdd with another ES20EARS. The check completed with 4 errors. I have attached a syslog with the full output, however the lines involved are: Jun 21 01:19:04 R2D2 kernel: md: recovery thread checking parity... Jun 21 01:19:04 R2D2 kernel: md: using 1152k window, over a total of 1953514552 blocks. Jun 21 01:19:11 R2D2 kernel: md: parity incorrect: 16240 Jun 21 01:19:11 R2D2 kernel: md: parity incorrect: 19552 Jun 21 01:19:11 R2D2 kernel: md: parity incorrect: 19560 Jun 21 01:19:11 R2D2 kernel: md: parity incorrect: 19568 Jun 21 09:08:45 R2D2 kernel: md: sync done. time=28181sec rate=69320K/sec I'm a little concerned, however I understand that the updates occured to correct parity information. What worries me is that there was bad parity to start with, and my data drive I upgraded may not have rebuilt cleanly in the first place. I have the old drive and have considered doing an initconfig with it, and starting the process from scratch, but wanted to get some feedback from the gurus here first Any ideas? Am I being overly paranoid? edit - added smartctl outputs for each drive unraid.txt smart.txt
June 21, 201115 yr I have seen similar behavior on my last three drive upgrades, in all cases I had done a parity check (that passed without error) before the upgrade and then stopped the array, then de-assigned the old drive and assigned the new drive (which was already in the system standing idle after having been pre-cleared). Here are the log fragments: 17-Mar-2011 Mar 17 08:08:01 saturn kernel: md: parity incorrect: 32048 Mar 17 08:08:01 saturn kernel: md: parity incorrect: 32056 9-June-2011 Jun 9 08:29:24 saturn kernel: md: parity incorrect: 17008 Jun 9 08:29:24 saturn kernel: md: parity incorrect: 17016 Jun 9 08:29:24 saturn kernel: md: parity incorrect: 27720 19-June-2011 I started a nocorrect parity check after the rebuild and immediately got 4 errors: Jun 19 06:23:20 saturn kernel: md: parity incorrect: 28496 Jun 19 06:23:20 saturn kernel: md: parity incorrect: 28504 Jun 19 06:23:20 saturn kernel: md: parity incorrect: 41112 Jun 19 06:23:20 saturn kernel: md: parity incorrect: 42856 Note that these errors (like yours) are being reported at very low block numbers, and (as in your case) a file system resize would have take place because I was increasing the size of these drives at the time. I suspect that when the file system is resized there is a glitch in the parity coverage and so the parity is incorrect (and not the data on the drive). But that's just a wild guess! I am using unRAID version 4.5.6. Regards, Stephen
June 21, 201115 yr Looks like the same issue is being discussed here: http://lime-technology.com/forum/index.php?topic=13560.msg128495#msg128495 Regards, Stephen
June 21, 201115 yr Author Thanks, I seem to have missed that thread in my quick search. I have one more upgrade yet to complete (sde: WD15EARS to WD20EARS), if you would like me to do anything before I start it later tonight, please let me know. After the upgrade I plan to add both old WD15EARS drives as new drives to the array (zeroing out the MBR and then re-adding them).
June 21, 201115 yr After the upgrade I plan to add both old WD15EARS drives as new drives to the array (zeroing out the MBR and then re-adding them). Why didn't you simply add the 2TB to your array, and then move the datas from the old drive to the new one if necessary? Your array will be safe all the time (no rebuilding process), less time, less power cycle, etc... Anyway, those sync errors are at least annoying. About parity check errors : Any more than zero is bad. That's my opinion too.
June 22, 201115 yr Author Why didn't you simply add the 2TB to your array, and then move the datas from the old drive to the new one if necessary? Your array will be safe all the time (no rebuilding process), less time, less power cycle, etc... Anyway, those sync errors are at least annoying. Because then I'd have to fiddle around with fixing up shares/split levels etc, which I don't have time to do. It's much easier for me to swap a drive or kick off a parity check then go to work or go to sleep, and do the next step later when I'm back at home. The point is that it *should* work. I know that IT doesn't always work the way it should (I work with geographically distributed FC/SSD enterprise storage in the multi-petabyte range) but it's a risk that I take. I have backups of the important stuff if it totally dies. In any case, I'm running MD5 checks on all the files on all volumes, from MD5 hashes stored when the files were moved to the array, and so far they have all passed clean. Odd.
June 22, 201115 yr Limetech is currently examining this bug. So far, there have been no reports of data corruption. It appears to be a parity flaw.
July 11, 201114 yr I have just done my first parity check about month after doing a couple of 1.5->2TB rebuilds... at the time of the rebuilds, no (0) parity errors were reported, but I think this may be the first time I have done a standalone parity check exercise since then... (fyi, I usually run a check monthly and have never seen an error before) For the first time ever, I have got 4 parity errors:- Jul 10 18:00:30 Tower kernel: md: recovery thread woken up ... Jul 10 18:00:30 Tower kernel: md: recovery thread checking parity... Jul 10 18:00:30 Tower kernel: md: using 1152k window, over a total of 1953514552 blocks. Jul 10 18:00:31 Tower kernel: md: parity incorrect: 13872 Jul 10 18:00:31 Tower kernel: md: parity incorrect: 13880 Jul 10 18:00:31 Tower kernel: md: parity incorrect: 13896 Jul 10 18:00:31 Tower kernel: md: parity incorrect: 13904 Otherwise, there are no other obvious errors and SMART is all good (still). Can anybody comment:- a) does this look like the same issue already discussed in this thread b) is this the 'housekeeping' area mentioned (the detection happens almost immediately after kicking off parity) - how big is this area and how do you know its extent? c) can I use these block numbers to verify any particular file/files? d) is there any useful info I can send to Tom (how/where?) to help him investigate? I am using unRAID Server Pro v4.7 thanks in advance!!
July 11, 201114 yr This is a known problem that Tom is working on. The most recent 5.0 beta includes a fix, but introduced some problem, so I am not recommending going there yet. The few sync errors you got, early in the disk, are in the RFS "housekeeping" area and are easily fixed with a correcting parity check. If you haven't done that, it would be the next step. Then wait for 4.7.1 (or 5.0 non-beta) to come out and you should not get parity sync errors on future disk rebuilds.
July 11, 201114 yr Thanks for the clarification... feeling much better about the data now - phew! I ran the parity check from the main unRAID menu, so I presume that did the corrections already - that's the default behaviour right? As for this housekeeping area, can you tell me how big it is (how can I recognize this area in future from the numbers in the log)? Also, is the nature of this bug simply that the parity data has fallen out of step with what's on the actual data disks (hence a parity correction fixes the situation)? Thanks again for the reassuring news. EDIT - still a little confused after finding some details in the 5.0-beta8c thread - if the nature of this bug is that there is a potential that data on a drive being rebuilt was not written, then wouldn't subsequent parity correction simply update the PARITY drive to agree with the MISSING data (or am I totally misunderstanding the parity process)??
July 12, 201114 yr Thanks for the clarification... feeling much better about the data now - phew! I ran the parity check from the main unRAID menu, so I presume that did the corrections already - that's the default behaviour right? As for this housekeeping area, can you tell me how big it is (how can I recognize this area in future from the numbers in the log)? Also, is the nature of this bug simply that the parity data has fallen out of step with what's on the actual data disks (hence a parity correction fixes the situation)? I am not sure exactly how big the housekeeping area is. But parity errors in the housekeeping area happen nearly immediately after starting a parity check. (1-2 quick refreshes). Thanks again for the reassuring news. EDIT - still a little confused after finding some details in the 5.0-beta8c thread - if the nature of this bug is that there is a potential that data on a drive being rebuilt was not written, then wouldn't subsequent parity correction simply update the PARITY drive to agree with the MISSING data (or am I totally misunderstanding the parity process)?? I think he is referring to the write not happening to the PARITY disk. And if parity is corrupt it could mean an imperfect reconstruction of another disk (were it to fail). That is why it is good that you corrected the parity error. No worries.
July 12, 201114 yr @queeg - I don't think so - my system is a MD-1510/LI Media Storage Server direct from lime-tech, and I don't think it uses those... So I ran reiserfsck on my 3 newest (rebuilt from 1.5TB originals) 2TB drives found this in the syslog on the 2nd drive I tested... is the "4 of them are zero" line important/significant/a problem?? Will read-only check consistency of the filesystem on /dev/md1 Will put log info to 'stdout' ########### reiserfsck --check started at Mon Jul 11 20:31:06 2011 ########### Replaying journal: Replaying journal: Done. Reiserfs journal '/dev/md1' in blocks [18..8211]: 0 transactions replayed Checking internal tree..finished Comparing bitmaps..finished Checking Semantic tree: finished No corruptions found There are on the filesystem: Leaves 426828 Internal nodes 2564 Directories 1921 Other files 11262 Data block pointers 430523294 (4 of them are zero) Safe links 0 ########### reiserfsck finished at Mon Jul 11 21:13:05 2011 ########### thanks again in advance, m * EDIT/UPDATE that drive is the only one of the three tested to report a non-zero number for the value in question...
July 12, 201114 yr Are your hard drives connected to a AOC-SASLP-MV8 by any chance? In my case they are. Stephen
July 14, 201114 yr since I still have the 1.5TB drive(s) which I believe are untouched since replacing them esp. including the for which the 2TB replacement is giving the "4 of them are zero" msg, is there an easy way to compare the files - can I mount the old drive without disturbing my current array and run some sort of file-by-file comparison? fwiw I haven't written any data to my array since getting the Parity correction notification thanks m [EDIT] - nevermind, I've figured it out and am running some 'diff' scripting right now... I'll let you know if I do or don't turn up any data-loss issues
July 15, 201114 yr all files on "that" disk diff'd without incident... so far so good - no data loss for that one at least...
July 16, 201114 yr no data loss on the 2nd of the 3 drives I rebuilt 1.5->2TB unfortunately, I already recommissioned the 3rd, so I'll never know about that one can ANYBODY comment as to the meaning/significance of the "Data block pointers"/"4 of them are zero" line from my log excerpt???
September 6, 201114 yr Just upgraded 2 more drives 1.5TB -> 2TB absolutely NO writes to these drives during the updates (fully abstained here!, upgraded them directly 1 after the other...) yet STILL I got these errors IMMEDIATELY post-upgrade on a parity check... Sep 5 20:29:24 Tower kernel: md: recovery thread woken up ... (unRAID engine) Sep 5 20:29:24 Tower kernel: md: recovery thread checking parity... (unRAID engine) Sep 5 20:29:24 Tower kernel: md: using 1152k window, over a total of 1953514552 blocks. (unRAID engine) Sep 5 20:29:25 Tower kernel: md: parity incorrect: 20888 (Errors) Sep 5 20:29:25 Tower kernel: md: parity incorrect: 20896 (Errors) Sep 5 20:29:25 Tower kernel: md: parity incorrect: 20904 (Errors) any ideas why and if there's any damage here??? - reassurance is good! (is this still in the 'maintenance/housekeeping' area - it showed up right at the beginning of the check again)? wish the fix (if it exists) would be released to the (stable) 4.7.x branch!! m
September 6, 201114 yr Just upgraded 2 more drives 1.5TB -> 2TB absolutely NO writes to these drives during the updates (fully abstained here!, upgraded them directly 1 after the other...) yet STILL I got these errors IMMEDIATELY post-upgrade on a parity check... Sep 5 20:29:24 Tower kernel: md: recovery thread woken up ... (unRAID engine) Sep 5 20:29:24 Tower kernel: md: recovery thread checking parity... (unRAID engine) Sep 5 20:29:24 Tower kernel: md: using 1152k window, over a total of 1953514552 blocks. (unRAID engine) Sep 5 20:29:25 Tower kernel: md: parity incorrect: 20888 (Errors) Sep 5 20:29:25 Tower kernel: md: parity incorrect: 20896 (Errors) Sep 5 20:29:25 Tower kernel: md: parity incorrect: 20904 (Errors) any ideas why and if there's any damage here??? - reassurance is good! (is this still in the 'maintenance/housekeeping' area - it showed up right at the beginning of the check again)? wish the fix (if it exists) would be released to the (stable) 4.7.x branch!! m This is a known issue. Has been fixed in the later 5.0 betas. Previously Tom indicated that he would release a 4.7.1 to address it, but now appears we are heading for a 5.0 release, so 4.7.1 may not happen. To correct your issue, simply run a correcting parity check. If all of the parity errors are in the housekeeping area at the beginning of the disk, then you can run the correcting parity check for just a short time until all of the parity sync errrors are corrected, and then stop it (or run it to completion if you like). Afterwards, I'd recommend another short parity check to confirm all of the sync errors are fixed.
September 6, 201114 yr thanks bjp999 you say "If all of the parity errors are in the housekeeping area at the beginning of the disk" - how do I know that is the case - I have shown you the log so what does that tell you?? I thought from previous reports that the bug was due to a race condition/conflict in writing to the array whilst upgrading??? (which I explicitly avoided doing) I am running the parity check now, so it will be correcting it regardless of where it exists... can I assume at the very least that since I didn't write to the array during the upgrades (migrating disks to bigger disks) that my data is safe? thanks, hopefully 5.0 isn't far off...
September 6, 201114 yr Yes - 20,000 is a very low numbered parity block. Sync errors should occur within seconds of starting the parity check. These symptoms are very consistent with the known bug. You should be good to go after you correct these parity errors.
September 6, 201114 yr thanks bjp - as mentioned, yup indeed these errors occurred within seconds of starting the parity check (and I've had none since...) as from my previous posts on earlier upgrades, I had similar experiences and no data loss I am still unsure as to:- 1) what is the block 'range' for the SAFE region in the 'housekeeping' area 2) what events lead to this error (I thought I knew, but still ran into it) in the meantime, I guess the only thing to do is wait for 5.0 thanks for the help and support
September 6, 201114 yr The bug results in parity not being updated in some specific situations where there are multiple updates to the same block. The rebuilt data disk is not affected. So your data rebulid worked fine. All of your data is intact. It is just parity that is wrong in a couple of spots. The risk is if you needed to rebuild ANOTHER disk. If you did that without correcting parity first, the rebulid would be incorrect in the locations where parity sync errors occurred. If those blocks contained actual data, then data would be corrupted on the rebulit disk and you'd have no way of knowing what files were affected. Buf if those blocks were in the housekeeping area, chances are extremely high it would not have any consequence. This is because most of the housekeeping area is buffer area used to journal disk writes. And on a properly unmounted disk, that buffer area has already been emptied, so it contains no useful data. I cannot tell you exactly the size of the housekeeping area. It may vary based on disk size. But parity sync errors that occur during the first several seconds on a parity check are in that area.
September 6, 201114 yr thanks for the detail - it gives me some insight for the future - although that should be the last of the upgrades until we get 3TB support in the main release :-) I did build (expand) 2 in a row so I don't know for which one of the two the errors crept in I have the original drives, so I can do a file-by-file compare today when I find the time from what you've said, these 3 errors occurred 1 second into the check, so should not affect the 'data' i.e. the files themselves - fingers crossed
September 7, 201114 yr just diff'd all files from the 2nd of the 2 disk expansions/rebuilds all files identical to originals :-) *EDIT* and did the same for 1st disk - again, all files OK
Archived
This topic is now archived and is closed to further replies.