July 22, 200817 yr Following a power outage, unRaid automatically started a parity check that reported way too many errors (over 10,000), the syslog is attached (syslog220708). I rebooted and re-did the parity check and although it is still running, after 25% it already shows 3000 sync errors. There are no read errors displayed. How should I proceed to isloate the problem and avoid data loss?
July 22, 200817 yr Following a power outage, unRaid automatically started a parity check that reported way too many errors (over 10,000), the syslog is attached (syslog220708). I rebooted and re-did the parity check and although it is still running, after 25% it already shows 3000 sync errors. There are no read errors displayed. How should I proceed to isloate the problem and avoid data loss? Let it finish the parity check.
July 22, 200817 yr tzotz - Can you give a chronology of your array: - When it was originally set up? - How many disks were in the array at that time? - What version of unRAID? - Did you add any drives to the array? - When? - What version of unRAID? Thanks for any details you can provde!
July 23, 200817 yr It looks like the initial parity check had completed before you attached the first syslog. Is this true? Are you saying that a subsequent parity check has found an additional 3000 errors (so far)? If so, and you let it complete, pretty sure it will be proper to let it do a parity calc one more time. If a third parity check finds additional errors, then there is a hardware issue we need to track down... since once parity was completely calculated, it should not be wrong on a subsequent run. Most suspect would then be memory, but it could be one of the disks or cables. What memory are you using? Did you set the voltage for it on the motherboard BIOS? Joe L.
July 23, 200817 yr I am experiencing the same thing, almost. The symptoms are the same, a parity disk that says it's ok, but multiple full parity checks have hundreds of thousands of sync errors reported. I noticed it after a drive delayed on spin up, and unraid failed it. The disk passes the full drive manufacturers test with no errors, and works fine, the data on it is fine. I replaced the disk in the unraid array with another drive, and it rebuilt it fine, which is strange since the parity checks show so many errors. Every subsequent check shows hundreds of thousands of sync errors, but no other problems seem to be apparent. All my data seems to be intact, and forcing a drive fail so it has to work from parity seems to work fine as well. Memtest runs for hours with no errors, and no other drive errors are appearing in the syslog. I'm stumped.
July 23, 200817 yr Author Thank you guys for your help I really appreciate it. I'll try to answer you in order: Can you give a chronology of your array: - When it was originally set up? - How many disks were in the array at that time? - What version of unRAID? - Did you add any drives to the array? - When? - What version of unRAID? Thanks for any details you can provde! The array was originally set up between Aug and November 2007 There were 3 disks there at that time the unRaid version was 4.1 or 4.2.1 I added a fourth drive around Feb 2008 The array version did not change remained 4.2.1 I switched to 4.2.4 in May 2008 and to 4.3.3 only a couple of days ago and I can't recall having seen so many sync errors (if any) before 4.3.3 - it could be but I seriously doubt it, I would've noticed.. It looks like the initial parity check had completed before you attached the first syslog. Is this true? Are you saying that a subsequent parity check has found an additional 3000 errors (so far)? If so, and you let it complete, pretty sure it will be proper to let it do a parity calc one more time. If a third parity check finds additional errors, then there is a hardware issue we need to track down... since once parity was completely calculated, it should not be wrong on a subsequent run. Most suspect would then be memory, but it could be one of the disks or cables. What memory are you using? Did you set the voltage for it on the motherboard BIOS? I did let the first parity check complete, the log says (at the bottom): md: sync done. time=6424sec rate=48656K/sec md: recovery thread sync completion status: 0 After the second parity check completed with as many errors as the first, I did initiate a third one but stopped it when I saw that errors pile up quickly. I'm using Kingston memroy, I don't recall the model ( the array is down now) but it was the one approved by the MB manufacturer (Intel). I didn't touch any voltages or any BIOS settings (other than the ones I'm supposed to set such as boot order) I am experiencing the same thing, almost. The symptoms are the same, a parity disk that says it's ok, but multiple full parity checks have hundreds of thousands of sync errors reported. I noticed it after a drive delayed on spin up, and unraid failed it. The disk passes the full drive manufacturers test with no errors, and works fine, the data on it is fine. I replaced the disk in the unraid array with another drive, and it rebuilt it fine, which is strange since the parity checks show so many errors. Every subsequent check shows hundreds of thousands of sync errors, but no other problems seem to be apparent. All my data seems to be intact, and forcing a drive fail so it has to work from parity seems to work fine as well. Memtest runs for hours with no errors, and no other drive errors are appearing in the syslog. I'm stumped. Same here, including the suffix I will try to disconnect the disks (luckily there are only 4) and move the array to another platform (if I find one that will support it) and report back In the meantime how do you guys suggest to tackle the problem? The array seems to work but if a disk fails (other than the parity) then I'm in deep ----, so I'm pressed to resolve this quickly. Thanks again for your response, I know you feel my pain.
July 23, 200817 yr I feel your pain. I would start with running the memory test through several full cycles. If it is flaky, then all bets are off. Joe L.
July 23, 200817 yr I also feel you pain! Wish I had a magic bullet. Below find some misc facts that I hope may help to resolve this issue: 1. Any time unRAID reports a parity error via the Web console, the error has already been fixed. By fixed I mean that parity, for the block in question, has been rewritten. So if you run a parity check and get 100 parity errors, and then run another parity check right afterwards, you shouldn't get any parity errors. 2. Parity errors cause the sector number for each failed parity condition to be written to the syslog. A problem can occur if there are so many parity errors that the log gets huge and fills up all available space on the OS ramdisk. When this fills up, the system likely crashes. This does not undo the parity corrections that have been made, and a reboot will restart the parity calculation, and eventually correct parity on the remainder of the disk. This is an annoying circumstance, but typically not a problem. 2. In general, unRAID "trusts" the data disks more than it trusts parity. So if a parity error does occur, unRAID assumes that it is parity, and not one of the data disks, that is wrong. All corrections are applied to the parity disk. 3. There are only two times that the parity data is used by unRAID to reconstruct data - (1) when a disk is being reconstructed; and (2) when a bad sector is detected. At these times, all of the disks (including parity) are read to reconstruct the data to be written to the target disk. Other than that, parity can be completely messed up and a user would not know unless they did a parity check. (No red balls, no loss of performance, no corrupted data - nothing in the syslong - NO CLUE). Since parity is so important for data reconstruction, it is each user's responsibility to ensure that parity is correct by running parity checks periodically. 4. To add a disk to an array, unRAID kind of cheats (cheats in a good way). What it does is fills the entire new disk with binary zeros. A zero value will not impact parity. Once the new drive is full of zeros, unRAID can silently slip it into the array without needing to rebuild parity. Several disks can be added simultaneously in this way. 5. A bug was introduced in version 4.3 beta 4 (April 1, 2008) where the code to fill a new disk with binary zeros didn't run properly. This bug was fixed in 4.3 Final (June 12, 2008). If someone added a disk to their array while using 4.3b4, 4.3b5, or 4.3b6 - their parity is likely wrong. The way to fix it is to run a parity check. It will likely find (and correct) a bunch of parity errors the first time it is run, and then the array will be integral again. There have been several occurrences of this reported in the forums. (Note this bug only occurred in beta versions, not in a "final" version, but still must be corrected). 6. In general, unRAID does an excellent job maintaining parity. With the exception of this clear disks bug, no recent bugs have been reported. The issue in this thread may ultimately be another bug, or may also be a hardware problem (like a memory error) or something else yet to be discovered. 7. There are ways of writing directly to a disk on the unRAID server that bypass the parity calculation. If you are running commands directly on the console or via telnet, that do device level output, parity could be inadvertently bypassed parity and creating parity problems. This is unlikely (most people that know enough to do this type of operation, also know how to use it wisely), but is a possibility. If you are running any system tools (e.g., a defragmenter), this could be causing problems. 8. I have been on a personal crusade, of sorts, to get users to run parity checks periodically. Parity checks cause each and every sector on each and every disk to be read. If a bad sector is encountered, a protected unRAID array can fix the problem. But if a drive were to actually fail in the array, and a user were trying to rebuild it, then unRAID has zero ability to correct these types of sector errors. So performing routine parity checks is a good thing to do as a maintenance activity, even if parity is perfect. Otherwise, bad sectors will develop that won't be discovered until a user is recovering the video of their kid's first birthday party! Of course parity checks also check parity - and can help identify issues like what is being reported in this thread. This is a good thing, as it can be corrected now. If users wait until a true failure has occurred and only THEN learn of the problem, it is too late! When users report these massive numbers of parity errors, I usually immediately think about #5 above. But based on your timeline, you never added a disk when using one of the affected beta versions. Please confirm. Joe L. is particularly good at helping users figure out even the most challenging problems. I'd recommend doing what he suggests, even if you think you have already done it. If he can't figure it out, Tom will likely need to get involved. Your syslogs provide a window into your system that will likely be requested as you do different things. I suggest starting to keep them before each reboot until this is resolved. Good luck!
July 23, 200817 yr Is there a command line we can run at the console to send the syslog to an external device or share? I really don't have enough space on my flash to save my syslogs, as they are huge, and my flash drive is relatively tiny. The last syslog was almost 70MB, and my flash is only 128MB or 256MB, I forget which. If I could send the syslog to my cache drive that would work too.
July 23, 200817 yr Author I'm going to run MEMTEST for a good number of hours and then report back. Will I run into any unrecoverable problems trying to run with 4.2.4 just to make sure the problem isn't 4.3 dependent? Thanks.
July 23, 200817 yr Is there a command line we can run at the console to send the syslog to an external device or share? I really don't have enough space on my flash to save my syslogs, as they are huge, and my flash drive is relatively tiny. The last syslog was almost 70MB, and my flash is only 128MB or 256MB, I forget which. If I could send the syslog to my cache drive that would work too. The only issue you would have is if you were one of those with the longer drive model numbers that were truncated to 16 characters in versions prior to 4.3. That would show up as a set of drives that all mismatch the ones in your existing configuration. Since you made no mention of that, odds are all of your hard disks have model numbers shorter than 16 characters. If the memory test shows anything, please list your motherboard model, and the specific model of memory chips you are using. It might help to figure out if the BIOS options are set appropriately. As far as saving all the syslogs. at the console, type: cd /boot gzip -v syslog*.txt The files will be compressed to a small fraction of their size and a .gz suffix appended. (gunzip syslogxxxxx.gz will uncompress. ) If you have already attached the syslog to this thread, you can copy it elsewhere, or delete it, as you desire. If you have a cache drive, you can copy the syslog there cp /var/log/syslog /mnt/cache/syslog_date_.txt I'll be interested to see if a subsequent parity check results in the same sectors being marked as invalid. (Do you have random errors, or errors in a sector range, or errors on a specific bit position? and I don't even know if we have enough information to determine that anyway... everything is a clue at this point.)
July 23, 200817 yr Author Ran MEMTEST for 7 hours without a single error. Booted the array with unRaid 4.2.4, ran "Check Parity" and got errors on the first (as expected) and on the second try. I'll be interested to see if a subsequent parity check results in the same sectors being marked as invalid. (Do you have random errors, or errors in a sector range, or errors on a specific bit position? and I don't even know if we have enough information to determine that anyway... everything is a clue at this point.) I will now make two consecutive Parity runs to check if errors are random. After that, I will take down drives one by one to see if any of them is causing this. It will take some time (hours). I will post result later.
July 23, 200817 yr Ran MEMTEST for 7 hours without a single error. Booted the array with unRaid 4.2.4, ran "Check Parity" and got errors on the first (as expected) and on the second try. I'll be interested to see if a subsequent parity check results in the same sectors being marked as invalid. (Do you have random errors, or errors in a sector range, or errors on a specific bit position? and I don't even know if we have enough information to determine that anyway... everything is a clue at this point.) I will now make two consecutive Parity runs to check if errors are random. After that, I will take down drives one by one to see if any of them is causing this. It will take some time (hours). I will post result later. This is looking like a hardware problem of some type. 4.2.4 was one of the most stable versions of unRAID, and if you are getting parity checks in 4.3.3 and in 4.2.4, it seems quite likely that this has nothing to do with unRAID. The problem could be with your motherboard, memory, hard drives, hard drive cabling, backplane or something external, like electrical interference. Memory is a common culprit, but sounds like you've ruled that out. Hard drive cabling is a surprisingly common problem, and easy to change. I've seen several apparent drive problems turn out to be problems with the backplane. Tom warns about one in particular (see the Best of link in my sig and look at the "Hail to the Chief" section) Motherboards can fail in many different ways, and can be very hard to diagnose without trying another. Some people have reported certain ports on their MB doing bad - something I have never seen personally. (I did have a problem that appeared to be some bad ports but a BIOS upgrade fixed the problem). Unless you have access to another one, MB failure is usually the last thing you suspect, after you've eliminated all the other possibilities. Electrical interference and other external factors are usually obvious. I would suggest you run "smartctl". See instructions on the Troubleshooting page (see link in my sig). I'd run it on each drive and post the results and I can see if any of the drives look like they are going (or have gone) bad. If that doesn't point you in a direction, I would recommend that you remove all drives from the array except one + parity. (You'll need to press the dreaded Restore button - read about it under "Best of" link, "Newbie" section). If you can reliably build and check parity with one data drive, add another data disk, press restore again, and rebuild parity again. Repeat until adding another drive causes parity problems. The problem will have something to do with adding that drive. The key with problem determination, IMO, is to get to a very basic configuration that works and start adding things back until it breaks. Starting with something that doesn't work and trying to go backwards to something that does work is not nearly as effective!
July 23, 200817 yr Apologies to tzotz for horning in on the thread, but my problems just sound so similar. I'm going to try to attach a syslog with 2 consecutive sync runs, it's now on the third run and looks to be on the same path. Scratch that, it's WAY too big to post, compressed it's almost 2.5MB. h__p://ctcomputers.net/logs/syslog_080723_.rar
July 24, 200817 yr Apologies to tzotz for horning in on the thread, but my problems just sound so similar. I'm going to try to attach a syslog with 2 consecutive sync runs, it's now on the third run and looks to be on the same path. Scratch that, it's WAY too big to post, compressed it's almost 2.5MB. h__p://ctcomputers.net/logs/syslog_080723_.rar Wow! That is a lot of parity errors. The parity errors seem to be in different parts of the disk with each check. There are long strings of consecutive (I think) errors - like every 8 sectors (4K). See my last response to tzotz. I think you've got hardware problems too. Maybe the same as him, maybe not. One thing I didn't mention is to make sure the motherboard BIOS is up to date.
July 24, 200817 yr Apologies to tzotz for horning in on the thread, but my problems just sound so similar. I'm going to try to attach a syslog with 2 consecutive sync runs, it's now on the third run and looks to be on the same path. A first comment, your NTP appears to be misconfigured, the following lines appeared in your syslog. ntpd_initres[2107]: host name not found: files ntpd_initres[2107]: couldn't resolve `files', giving up on it A comment as to your 3 terabyte Seagates, they are linking at 1.5 Gbps, instead of 3.0 Gbps. That usually indicates that they still have the SATA limiting jumper installed. See the Improving unRAID Performance page for help. The rest of the hardware and driver setup is fine, although there are some odd messages concerning your 2 Promise controllers. I don't know if they are significant, but I don't think so. They are probably additional reporting added to the newer version of the Promise driver included with the newer kernel. You stopped the array, re-assigned the parity drive, and started a parity sync, which finished fine, and with what appears to be normal performance, considering the older IDE drives. Then you started a parity check, stopped it when you saw all of the errors piling up, then started another parity check, with many more errors, and you let it finish. I don't have time right now to fully analyze the error patterns, but there ARE some interesting patterns, that *may* be indicative of something. At this point, it does appear to be a hardware issue, but I don't have a single hardware error to back me up! At first glance, the sequences of parity errors do not appear to overlap or coincide, between the 2 runs. But they definitely quit right at the 300GB point, which is the last of the drives on the Promise controllers. You have 4 300GB drives, Maxtors, attached one per channel to the 2 Promise cards, which is normally a very good idea, but in this case may not be working correctly. In the past, there have been problems occasionally with multiple Promise controllers installed, but I don't know if the 2 different controllers you have are known for any compatibility issues. *If* there are compatibility issues, then I would have expected to see hardware issues reported, and there are NONE! So my credibility here is very low, and you can feel free to ignore my suggestions. However, the 300GB limit on the errors leads me to suggest eliminating one or both of the Promise controllers, just for testing. If you eliminate one controller and add all 4 drives to the remaining one (after changing jumpers on 2 drives, a nuisance), that would eliminate one suspect. When I get more time, I may have more analysis and ideas...
July 24, 200817 yr About the NTP, it's not misconfigured, my NTP server machine just happened to be shut down at that point. The 300GB boundary for the errors is VERY interesting, I saw huge numbers and gave up figuring out where they were. The promise controllers are quite old, and one is a RAID only model, it complains about no array every time I boot, forcing me to press escape to continue. I've been badly burned in the past by putting more than 1 drive on a channel, so I don't do that any more. I guess the next step is to unassign drives from the array and do a restore and a sync? If I unassign drives, the data on them will remain untouched, correct?
July 24, 200817 yr I guess the next step is to unassign drives from the array and do a restore and a sync? If I unassign drives, the data on them will remain untouched, correct? correct.
July 24, 200817 yr Author I'll be interested to see if a subsequent parity check results in the same sectors being marked as invalid. (Do you have random errors, or errors in a sector range, or errors on a specific bit position? and I don't even know if we have enough information to determine that anyway... everything is a clue at this point.) Attached log of two consecutive Parity Check runs. The number of errors is quite different (14k VS 11K) so it would seem to be random. I will now remove disks from the array one by one. I'll also follow up on Joe's and BJP's comments to see what I can find out, thanks guys. Until next time,...
July 24, 200817 yr Attached log of two consecutive Parity Check runs. The number of errors is quite different (14k VS 11K) so it would seem to be random. Nothing attached! Relook at my suggestions. Run smartctl on your drives, check you MB BIOS, and if everything is okay - go back to a 2 drive configuration and see if you can get that to work. It will save you a lot of pain rather than taking disks out one at a time!
July 24, 200817 yr Author Relook at my suggestions. Run smartctl on your drives, check you MB BIOS, and if everything is okay - go back to a 2 drive configuration and see if you can get that to work. It will save you a lot of pain rather than taking disks out one at a time! BJP, just to make sure I undestood: first run smartcrl on all disks if OK then do the following: 1) stop the array 2) unassign all disks except the parity and one disk 3) Press Restore 4) start the array 5) Check parity 6) if OK stop the array 7) assign another disk go back to step 3 No danger of data loss if I follow the above procedure right?
July 24, 200817 yr 1) stop the array 2) unassign all disks except the parity and one disk 3) Press Restore 3.5) Carefully examine the screen to make sure that the drives are assigned as you want them. If any of your drives show up as unformatted, do not start the array. 4) start the array 4.5) unRAID will rebuild parity on its own 5) Check parity (press Parity Check) 6) if OK no parity errors, stop the array 7) assign another disk go back to step 3 At step 7, you might decide to add a couple at a time to reduce the number of times you have to go through this.
July 24, 200817 yr Author Hi, I ran the smartcrl utility (output attached). The only thing I could easily discern is that there were error on SDB. Should I now just take that drive off line and try the restore or go the other way (remove all but one and add one at a time)? Again many many thanks for your time and help.
July 24, 200817 yr Hi, I ran the smartcrl utility (output attached). The only thing I could easily discern is that there were error on SDB. Should I now just take that drive off line and try the restore or go the other way (remove all but one and add one at a time)? Again many many thanks for your time and help. If you look at the errors, it appears that they happened long ago (at 9 hours, drive has not been on 358 hours). Sometimes a cabling problem at iniitial install can trigger these errors. Doesn't mean the drive is bad. I don't see any problem with any of these smartctl outputs. Drives look very good. Temps are good. No remapped sectors. Did you check for a BIOS update? If so, I think I'd try hooking up 1 disk + parity and see if you can get a successful parity rebuild and parity check. Use a disk other than SDB just to be safe (unless that is the parity disk).
Archived
This topic is now archived and is closed to further replies.