I am running the 5.0rc8a software.  Running a routine parity check in preparation for swapping out my parity drive for a new unit.  Ran a check 10 days ago, no errors.  Tonight for the first time, I am seeing errors during my parity check and I am very worried.  I usually run a non-correcting parity check, and I am 99% sure I unchecked the box again this time before starting the parity check, but $%^^&*((^ parity check is correcting errors and I have 5 so far - about half way through.  They seem to have been corrected. 

 

The errors seem to be in a cluster, detected about 2.5 hrs into the check, and within the same second, according to the log.  Since they have been corrected, I would like to identify the file(s) that have been affected - can someone please tell me how to determine the filename of the file(s) affected by the incorrect parity? 

 

I am also confused by the two web screens - the Mymain page says the parity check is correcting:

 

But the Unraid main seems to show that the "correct" box is unchecked and the errors uncorrected? 

 

I'm attaching a log file.  I'd really appreciate help on this - I am concerned about data loss.  Secondarily I am very upset that I am unable to run a non-correcting parity check, from the Unraid Main web page, even though I am pretty certain that I selected not to correct.

 

Thanks.

syslog-2013-01-05.txt

It's not correcting.

 

Jan  5 17:22:21 Repository kernel: mdcmd (408): check NOCORRECT

 

There is no way to determine why you are getting the errors unless you find a disk that is acting up and not reading correcting. No way to determine which file is affected either.

Thanks for pointing that out - I should have noticed that.  So this is something not right with the way Unmenu interacts with the rc, I suppose. 

 

I guess the plan of action is to finish this parity check, then run another non-correcting check, and if the errors persist, to correct them?  After that I can swap my parity disc.  I would feel much better if I could identify the file.

 

Thanks a lot for your help.

The server completed the parity check, finding no further errors.  I am rerunning the check now to verify the finding. 

 

I ran  smart reports on all of the drives, and all passed, but I would appreciate it if someone could take the time to look through them in case there is something I am missing. I attached them in a zip file.

 

Thanks!

Smart_Reports_20130106.zip

I am running the second parity check now, and I see the same exact five errors: 

 

Jan  5 17:22:21 Repository kernel: mdcmd (408): check NOCORRECT (unRAID engine)

Jan  5 17:22:21 Repository kernel: md: recovery thread woken up ... (unRAID engine)

Jan  5 17:22:21 Repository kernel: md: recovery thread checking parity... (unRAID engine)

Jan  5 17:22:21 Repository kernel: md: using 15360k window, over a total of 2930266532 blocks. (unRAID engine)

Jan  5 19:49:35 Repository kernel: md: parity incorrect: 1565565768 (Errors)

Jan  5 19:49:35 Repository kernel: md: parity incorrect: 1565565776 (Errors)

Jan  5 19:49:35 Repository kernel: md: parity incorrect: 1565565784 (Errors)

Jan  5 19:49:35 Repository kernel: md: parity incorrect: 1565565792 (Errors)

Jan  5 19:49:35 Repository kernel: md: parity incorrect: 1565565800 (Errors)

Jan  5 23:22:11 Repository unmenu-status: Exiting unmenu web-server, exit status code = 141

Jan  5 23:22:11 Repository unmenu-status: Starting unmenu web-server

Jan  6 01:08:07 Repository kernel: mdcmd (409): spindown 6 (Routine)

Jan  6 01:59:04 Repository kernel: mdcmd (410): spindown 1 (Routine)

Jan  6 01:59:45 Repository kernel: mdcmd (411): spindown 3 (Routine)

Jan  6 01:59:56 Repository kernel: mdcmd (412): spindown 7 (Routine)

Jan  6 02:00:17 Repository kernel: mdcmd (413): spindown 8 (Routine)

Jan  6 03:42:01 Repository kernel: md: sync done. time=37179sec (unRAID engine)

Jan  6 03:42:01 Repository kernel: md: recovery thread sync completion status: 0 (unRAID engine)

Jan  6 04:42:08 Repository kernel: mdcmd (414): spindown 0 (Routine)

Jan  6 04:42:08 Repository kernel: mdcmd (415): spindown 2 (Routine)

Jan  6 04:42:11 Repository kernel: mdcmd (416): spindown 4 (Routine)

Jan  6 04:42:11 Repository kernel: mdcmd (417): spindown 5 (Routine)

Jan  6 04:42:13 Repository kernel: mdcmd (418): spindown 9 (Routine)

Jan  6 04:42:14 Repository kernel: mdcmd (419): spindown 10 (Routine)

Jan  6 08:41:27 Repository kernel: mdcmd (420): check NOCORRECT (unRAID engine)

Jan  6 08:41:27 Repository kernel: md: recovery thread woken up ... (unRAID engine)

Jan  6 08:41:27 Repository kernel: md: recovery thread checking parity... (unRAID engine)

Jan  6 08:41:27 Repository kernel: md: using 15360k window, over a total of 2930266532 blocks. (unRAID engine)

Jan  6 11:09:07 Repository kernel: md: parity incorrect: 1565565768 (Errors)

Jan  6 11:09:07 Repository kernel: md: parity incorrect: 1565565776 (Errors)

Jan  6 11:09:07 Repository kernel: md: parity incorrect: 1565565784 (Errors)

Jan  6 11:09:07 Repository kernel: md: parity incorrect: 1565565792 (Errors)

Jan  6 11:09:07 Repository kernel: md: parity incorrect: 1565565800 (Errors)

 

I am not sure how to proceed - how do I know if it is the data on the parity drive or on the data drive which is in error?  Do I correct parity or do something else?

I took a quick look and didn't see anything in the SMART reports. Do the check a third time and see if the result duplicates again. If yes, then I would just correct parity. The consistent errors aren't indicating that a drive is acting flaky. Just keep an eye on it.

Thanks, Lionel.  I post an update tomorrow.

Update.  As I posted above, running a second non-correcting parity check resulted in exactly the same errors.  Based on that, instead of running a correcting parity check, I swapped out my parity drive as planned, and established parity yesterday.  I have begun a non-correcting parity check on the new parity drive.  I have not seen any errors yet and I will post a (final) update tomorrow and mark this issue closed.

 

I really appreciate your input!

Replacing parity has the same effect as a correcting parity check.

A follow-up.  I re-checked parity last night, testing the newly established parity drive.  This parity check resulted in no sync errors. 

 

I will be adding my old parity disk to my array as a data disk at some point in the future but keep it as a warm spare for now.  I believe my array is okay.

 

thanks very much for your advice, dgaschk, I appreciate it.

OK, i have just run another routine monthly parity check, and I see the exact same five errors.  The array has been rebooted twice since my last post.  Now I worry - and how do I find out where these errors are, maybe I have a bad disk.

 

Any thoughts?

 

Dec 27 13:46:18 Repository kernel: NTFS driver 2.1.30 [Flags: R/W MODULE]. (System)

Dec 27 13:46:38 Repository unmenu-status: Exiting unmenu web-server, exit status code = 141

Dec 27 13:46:38 Repository unmenu-status: Starting unmenu web-server

Dec 27 14:47:46 Repository kernel: mdcmd (805): spindown 6 (Routine)

Dec 27 15:04:17 Repository kernel: mdcmd (806): spindown 1 (Routine)

Dec 27 15:04:57 Repository kernel: mdcmd (807): spindown 2 (Routine)

Dec 27 15:04:58 Repository kernel: mdcmd (808): spindown 5 (Routine)

Dec 27 22:19:59 Repository kernel: mdcmd (809): check NOCORRECT (unRAID engine)

Dec 27 22:19:59 Repository kernel: md: recovery thread woken up ... (unRAID engine)

Dec 27 22:19:59 Repository kernel: md: recovery thread checking parity... (unRAID engine)

Dec 27 22:19:59 Repository kernel: md: using 8000k window, over a total of 2930266532 blocks. (unRAID engine)

Dec 28 00:29:47 Repository kernel: md: parity incorrect, sector=1565565768 (Errors)

Dec 28 00:29:47 Repository kernel: md: parity incorrect, sector=1565565776 (Errors)

Dec 28 00:29:47 Repository kernel: md: parity incorrect, sector=1565565784 (Errors)

Dec 28 00:29:47 Repository kernel: md: parity incorrect, sector=1565565792 (Errors)

Dec 28 00:29:47 Repository kernel: md: parity incorrect, sector=1565565800 (Errors)

Dec 28 05:45:06 Repository kernel: mdcmd (810): spindown 1 (Routine)

Dec 28 05:45:07 Repository kernel: mdcmd (811): spindown 3 (Routine)

Dec 28 05:45:07 Repository kernel: mdcmd (812): spindown 6 (Routine)

Dec 28 05:45:08 Repository kernel: mdcmd (813): spindown 7 (Routine)

Dec 28 05:45:09 Repository kernel: mdcmd (814): spindown 8 (Routine)

Dec 28 07:03:29 Repository kernel: md: sync done. time=31410sec (unRAID engine)

Dec 28 07:03:29 Repository kernel: md: recovery thread sync completion status: 0 (unRAID engine)

zip and attach SMART reports.

This may help.

 

http://lime-technology.com/wiki/index.php/FAQ#How_To_Troubleshoot_Recurring_Parity_Errors

 

Here is a zip file with the smart reports and my syslog.  Thanks for your help.  I am looking through the faq.

Smart_Reports_1229.zip

syslog-2013-12-29.txt

i think i should try the scripting suggestion listed in the fact, but I am not comfortable just copying what is there since I dont really know what I am doing and this is a pretty full server.  Can someone walk me through this? 

 

What do I do - create a txt file like this for each sdx? :

 

#!/bin/bash

LOG_DIR=/var/log/hashes

cd $LOG_DIR

 

for i in {1..5}

  do

    echo "Begin sdc for the $i time."

    dd if=/dev/sdc skip=156556570 count=200 | md5sum -b >> sdc.log

  done

exit

 

How do I execute it? 

One more thing.  In the syslog you will see that this last parity check clearly says "NO CORRECT" however, in the main display it says parity was checked two days ago and 5 corrections were written to the parity disk. 

 

I'm running another check now.  I will do the scripts testing if someone can help me with that.

This will test all disks. It takes 2 or 3 arguments. The first 2 arguments are required. They are skip, this value can be found in the log, and the count. The 3rd argument is optional. It can be used to restrict the set of test disks.

 

#!/bin/bash

if [ $# -lt 2 ]; then
echo "USAGE: $0 skip count [dev]"
echo "E.g., $0 156556570 200"
echo "E.g., $0 156556570 200 sd[a-c]"
exit
fi
DEV=[sh]d?
if [ $# -eq 3 ]; then
DEV=$3
fi
LOG_DIR=/var/log/hashes
mkdir -p $LOG_DIR
cd /dev

for d in $DEV
do
       if [ $(stat -c %G $d) == disk ]; then 
		for i in {1..5}
		  do
			echo "Begin $d for the $i time."
			LINE2=$LINE1
			LINE1=$(dd if=/dev/$d skip=156556570 count=200 | md5sum -b)
			if [ $i -gt 1 ] && [ "$LINE1" != "$LINE2" ]; then 
				echo "$LINE1 ERROR" >> $LOG_DIR/$d.log
				echo ERROR
			else 
				echo "$LINE1" >> $LOG_DIR/$d.log
			fi
		  done
		echo >> $LOG_DIR/$d.log
	fi
done		  
exit

 

Save this file on the server as "dtest.sh" and enter "chmod +x dtest.sh"

 

 

This may not reveal the problem however. It should reveal if a disk is not retuning a constant value. However the issue may only occur during an interaction of multiple components. For example, if the the error is introduced by the SATA card only during concurrent reading. Reading individual disks may succeed and reading from multiple, i.e., in a parity check, may fail (This has happened!). To determine the problem in the latter case, multiple parity checks are required on arrays of various sizes. E.g., if the system fails with all disks, then build any array with only half the disks and run parity check. Then repeat with the other half of the disks until the problem components are isolated.

 

 

One more thing.  In the syslog you will see that this last parity check clearly says "NO CORRECT" however, in the main display it says parity was checked two days ago and 5 corrections were written to the parity disk. 

 

I'm running another check now.  I will do the scripts testing if someone can help me with that.

 

 

This is a known bug. Corrections were NOT written.

Here is a zip file with the smart reports and my syslog.  Thanks for your help.  I am looking through the faq.

 

Disks 1,2 and 4 appear to have power issues. See here: http://forums.storagereview.com/index.php/topic/23324-power-off-retract-count-what-does-it-mean/

dgaschk,

 

I ran the script from the /boot folder.  it ran, but nothing  happened - should there be output to the screen?  There is nothing written to the syslog, either.

 

thanks,

Look in /var/log/hashes/

This may not reveal the problem however. It should reveal if a disk is not retuning a constant value.

 

 

Yes, that is why the link I provided had examples of kicking off reads for every disk at the same time.

Look in /var/log/hashes/

 

I dont see a /hashes directory in /var/log...

 

How are you running the script? Copy and paste the terminal session that shows it run.

Just as a matter of interest, when you found that you had sync errors, why were you reluctant to let UnRAID fix them?

 

When you replaced the parity disk and did a new parity sync, that had EXACTLY the same impact as simply running a correcting check ... except that you were "running at risk" while the new parity sync was in process -- whereas the array would have still been protected during a correcting check.

 

As for identifying which files are potentially impacted by sync errors ... it's possible to identify a "set of files" that could possibly be impacted, but it's FAR more likely that none of those files have errors and that the error is simply a sync error on the parity drive.  Personally, I never run non-correcting checks.  [Why would I NOT want my parity to be correct??]

 

You can read my detailed thoughts on this in the 2nd post here:

http://lime-technology.com/forum/index.php?topic=31020.0