Hi All,

 

This is my 2nd topic in the last week as the unRAID has been through some issues (Box is now stable, running 5.0.4, no bad disks, good parity 2 days ago). 

 

I parity checked 2 days ago with no errors.  Yesterday I came home to no power for whatever reason (I don't have a UPS yet but it was a clear day).  Power came back on; I started up my machines and the unRAID box.  After about 5 minutes, the WEBGUI wasn't coming up but I could Telnet into the box.  It was showing HD activity.  I hastefully tried to do a manual powerdown in Telnet.  It wasn't successful and ultimately I unmounted a few disks but it said some weren't mounted, etc.  Anyway, a hard power off had to occur when I believe the box came up and had initiated a parity check (I don't know what happened on first boot, likely everything was fine and I just had to wait a bit longer for WEBGUI).

 

The parity check completed with 60 sync errors.  I don't see any read errors in the log but the sectors that had the errors are spread out.  I was wondering if there was anyway I could check to see what data was effected and "test" it to see if it's corrupt.  Should I be worried about this? 

 

Here is truncated piece of log:

Dec 19 19:28:37 PetroNAS kernel: md: correcting parity, sector=87817552

Dec 19 19:37:47 PetroNAS kernel: md: correcting parity, sector=159118360

Dec 19 20:29:32 PetroNAS kernel: md: correcting parity, sector=565978800

Dec 19 20:33:52 PetroNAS kernel: md: correcting parity, sector=598843768

Dec 19 21:05:12 PetroNAS kernel: md: correcting parity, sector=812384432

Dec 19 21:42:43 PetroNAS kernel: md: correcting parity, sector=1045692616

Dec 19 21:42:43 PetroNAS kernel: md: correcting parity, sector=1045692808

Dec 19 21:42:46 PetroNAS kernel: md: correcting parity, sector=1046099800

Dec 19 21:42:47 PetroNAS kernel: md: correcting parity, sector=1046295368

 

As you can see, it's spread out all over.  Do I need to look at this command  reiserfsck from this thread : http://lime-technology.com/forum/index.php?topic=3309.0 (which btw has great explanation of parity).  Based on that thread, it seems like I shouldn't be worried since I didn't have any read errors but I just want some expert opinions and if there is some way I can check what data is across these sectors?

 

Thank you all for your help and time. 

 

Chris

 

 

syslog_parity.txt

Unfortunately there is no way to identify the files that are affected. 

 

This is why many believe in creating CRC checksums of all the files on the disks to protect against data 'bit rot'.    You can then periodically repeat this to see if any file checksums have changed.

Unfortunately there is no way to identify the files that are affected. 

 

This is why many believe in creating CRC checksums of all the files on the disks to protect against data 'bit rot'.    You can then periodically repeat this to see if any file checksums have changed.

 

Thanks for the reply.  Interesting...not what I was hoping for but I had a feeling.  In your opinion, should I be concerned?  I mean 60 sectors not matching the parity disk 2 days after a parity check that was perfect concerns me.  Nothing was being written to the disk at either time (the power outage or the hard reset when it was likely doing the parity check).

 

What is the best tool to map out CRC checksums in your opinion?  I have about 4TB of data, I imagine this mapping would take a long time and lots of CPU cycles?

 

Anyone else have any thoughts?

 

Thanks again.  I feel like I will always have a doubt now; I need to get a UPS urgently. 

 

Chris

I use the md5deep program that I install from unMenu to generate my checksums.

The best way to confirm that all is okay is to compare the data on UnRAID to your backups.

 

If you don't have backups, then there's really no way to know for sure.    I can say that the few times I've had sync errors in the last 5-6 years that I've used UnRAID, they have ALL been legitimate sync errors on the parity disc, and never an error in the actual data.    [i've run a complete compare against my backups every time that happened]

 

If you don't have a backup strategy, and/or don't think your data is important enough to bother with backups; then you can at least create checksums so you'll have a way to identify any files that have been corrupted.    I do this in addition to backups these days.    I use Corz excellent checksum utility:  http://corz.org/windows/software/checksum/    You simply install it on a Windows box;  then point to a share on the server; right-click; and select "Create checksums".    This will, of course, run for a long time -- but when it's done you'll have checksums for every file on that share.  Repeat for each share (or do it for each disk, whichever you prefer).    Then in the future, you can easily check to see if any files have changed by just right-clicking and selecting "Verify checksums".

I use the md5deep program that I install from unMenu to generate my checksums.

 

Thank you Bob, I will look at that. 

 

The best way to confirm that all is okay is to compare the data on UnRAID to your backups.

 

If you don't have backups, then there's really no way to know for sure.    I can say that the few times I've had sync errors in the last 5-6 years that I've used UnRAID, they have ALL been legitimate sync errors on the parity disc, and never an error in the actual data.    [i've run a complete compare against my backups every time that happened]

 

If you don't have a backup strategy, and/or don't think your data is important enough to bother with backups; then you can at least create checksums so you'll have a way to identify any files that have been corrupted.    I do this in addition to backups these days.    I use Corz excellent checksum utility:  http://corz.org/windows/software/checksum/    You simply install it on a Windows box;  then point to a share on the server; right-click; and select "Create checksums".    This will, of course, run for a long time -- but when it's done you'll have checksums for every file on that share.  Repeat for each share (or do it for each disk, whichever you prefer).    Then in the future, you can easily check to see if any files have changed by just right-clicking and selecting "Verify checksums".

 

Gary, Thank you as well.  I've seen your other posts about this.  I feel a little bit better, but, I don't have backups of EVERYTHING on the array but perhaps the stuff I do have backups of I'll do a check sum compare to be sure. 

 

I really appreciate all this.  I definitely need a better strategy as power outages on sunny days do happen occasionally and so does "operator error".  One thing that I liked about 4.7 vs. 5.0.4 is that when the array came up successfully, I'd get 2 very quick beeps from the PC speaker.  I'd say to myself, great, let me login to the WebGUI.  With 5.0.4 it seems this is gone so I really have no way to tell when the WebGUI is assessible besides hitting the IP a few times. 

 

Thanks again,

 

Chris

It won't be at exactly the same point in the process, but you can add "beep;beep;beep"  at the end of your GO script and you'll hear 3 beeps when it executes 

 

FWIW, I definitely consider a UPS a mandatory accessory for ALL of my computers ... especially my UnRAID servers 

When power is lost, data not yet flushed to the disks is lost.

 

The file-system on the data disks is a journaled file-system.  It is able to deal with the loss by re-playing the journaled.

 

The parity disk has no such mechanism to recover.  as you said, it is most likely to be in error.

It won't be at exactly the same point in the process, but you can add "beep;beep;beep"  at the end of your GO script and you'll hear 3 beeps when it executes 

 

FWIW, I definitely consider a UPS a mandatory accessory for ALL of my computers ... especially my UnRAID servers 

 

Thanks again.  I started looking at UPS's today.  It's been on the list for awhile, they aren't as expensive as I thought.  I'm looking at the APC Back UPS Pro 1500. 

When power is lost, data not yet flushed to the disks is lost.

 

The file-system on the data disks is a journaled file-system.  It is able to deal with the loss by re-playing the journaled.

 

The parity disk has no such mechanism to recover.  as you said, it is most likely to be in error.

 

Joe - what I'm hearing from you is you think I'm OK since there wasn't any read errors on the disks?  Could I have created this when I manually unmounted disks when the array was potentially already doing a parity check (the first reboot after the power loss).  In order to prevent an automatic parity recheck upon power loss, I'd basically have to set the array not to start on power up.  I'm not sure I want to do that, but is there way to not have it do an error correcting parity check automatically? 

 

Thanks again for your thoughts. 

 

Chris

There are a variety of things that could cause UnRAID to lose parity sync during power outages ... that's almost certainly what happened.    Simply stopping the parity-check-on-reboot shouldn't cause any additional issues -- although if you stopped it by manually unmounting the disks and not via the WebGUI that may have had some impact.

 

But if you simply add a UPS, you won't have any more incorrect shutdowns due to power loss .. so the whole issue will just go away      Get a quality unit with AVR ... the APC Back UPS Pro 1500 you're looking at is an excellent choice.