We had a power outage today.  The last activity on the unRAID box was over 12 hours BEFORE the outage, so I'm sure that all the drives were spun down.  Upon reboot it is running the parity check, and it had reported 61 sync errors by the time I brought up the Web GUI  (it is only 0.2% through the parity check).  I have now watched it for over 5 minutes (it's up to 1% complete) and there are no more sync errors being reported.  Is this normal?  Should i be worried?

 

Attaching screenshot and syslog.

 

(Note that it appears my clock got reset by the power outage, and therefore the timestamps in the log are not accurate.)

 

Notiice the writes on disk9 and parity.  disk9 was the recipient of the last data copied to the array.  (It was done via a "mv" command issued through a telnet session from a disk in the unRAID server that was not a part of the array.)

 

I believe that unix maintains a journal of pending writes near the beginning of the drive, and that after an unscheduled reboot, it will apply those transactions.  I am wondering if somehow those updates were applied to disk9 in a way that unRAID (parity) doesn't see and that this creates sync errors.  I am wondering if there will be more when it gets to the sectors where the last of the updates were made in the data area (somewhere around the 440G mark.

 

61 sync errors is somewhat worrying, but Parity Check's not only find problems, they fix them.

 

Your syslog troubles me.  All but Disk 7 and Disk 9 reported 2 transactions replayed, which is very normal after an improper shutdown, although a little troubling in itself.  But then I kept waiting to see Disk 7 and 9 report in, but instead got quite a list of parity errors, until finally both reported in.  Disk 7 took 5 seconds to replay 5 transactions, and Disk 9 'replayed 263 transactions in 146 seconds', which says to me that the Parity Check process clearly did not wait until ALL of the disks had finished mounting.  Those last transactions took a long time, because the computer was hard at work reading every drive for the parity checks.  Those log entries do not seem like just delayed log updates.

 

A journaling file system like ReiserFS is great for preserving data integrity, and dealing with incomplete writes, which makes it very good at handling power outages.  But transactions are all about protecting writes to the drives, not reads, so the parity drive has to be updated at the same time for each transaction.  I find it very difficult to believe that this could be a good thing, trying to mount a drive and write the last transactions and change the associated parity blocks, while at the same time checking parity.  Perhaps it does work, but sure sounds incredibly risky to me.

 

You are probably a little less paranoid than me, about operations like this, but if it were my system, I would feel happier after running reiserfsck on both drives, then running a second parity check.  But that's just me.

 

I suspect that you, like me, are wondering why ALL of the transactions and data had not been written to disk long before, long before the drives had spun down.  That does bother me, and I would like to hear from others any ideas why all buffers were not flushed and transactions written out within seconds after idling a bit.  It does not seem right.  Why hang on to an incomplete write 'just because you can'?  Flush it to disk and be done with it.

 

As to seeing more parity errors when it approaches the updated area on Disk 9, I wouldn't expect a problem, because parity blocks are updated at the same time as the data blocks.

 

Thanks RobJ!!!

 

I reread your post several times trying to figure out whether I am in danger of data loss or corruption.

 

What I think I hear you saying is ...

 

ReiserFS formatted drives keep a journal of writes on the disk, that are automatically replayed if the drive is shutdown improperly.  When I rebooted after the power failure, unRAID started checking parity BEFORE all the drives were able to complete processing their journals. The PCI bus was so busy that completing the journal writes took way longer than normal.  As a result, sections of the array were parity checked BEFORE the disk had replayed all its transaction - creating 61 sync errors. 

 

When there is a sync error, parity gets rewritten to match the data.  When the replayed transactions finally happened (after the parity check had already passed that spot on the disk), parity should have gotten updated again and all should be okay now.  And since all the logs have now been applied to my disk, all should be well for the rest of the disk. 

 

Long and short of it - I am probably in good shape, but it might be worth running reiserfsck and checking parity yet again after this parity check parity is finished.  I think I can just run parity check for a minute or two and then stop it, because it is only the first 0.2% of the disk where there might be a problem.

 

Perhaps Tom should put a delay between mounting the drives and starting the array after a dirty shutdown, to give the journals a chance to be get processed before engaging parity protection.  Then when the parity check starts, the disk should be accurate, not in some half updated state.

 

And then there is the question about why the I/O buffers weren't flushed to disk long before.  Would it be a good idea to run the "sync" command after completing an intensive "writing to the array" session?

 

-Brian

You 'translated' me very well, I'm afraid I am not a very good communicator.  One point, I don't believe we understand what's going on well enough to make any conclusions as to the cause of the parity errors.  But I agree, you are probably in good shape.

 

It does seem right that the parity thread should wait on completion of mounting and transaction processing.  The Linux 'sync' operation seems like core kernel stuff.  I don't understand why that is not happening automatically.  Buffers should be flushed as soon as there is idle time.

 

Your explanation was great!  I was just trying to figure out whether it meant that I had corruption or not.  I guess neither of us is completely sure, but it looks like there is a good chance that things are okay.

 

Parity check completed without further sync errors (as expected).

 

I did a quick parity check this morning, and got a single parity error.  I am posting the syslog again so that if Tom looks at this, he will have all the data.  I stopped the parity check after it got to 0.3% complete.

 

I am going to double check the files I recently copied and make sure they are okay tonight.

 

Thanks again RobJ!

A brief flicker of lights last night and bam, down went the computers!  Unbelievable.  I was in the middle of deleting a large number of files on the unRAID server.  There were 43 sync errors, a 2nd check gave 0 errors. The difference this time is that about 1/2 the errors occured in the beginning of the disk, and the rest occurred throughout in different spots in the data area.

 

Tom, if you are reading - how worried should a person be if there are sync errors after a power outage?  Based on what RobJ said, and research, reiser formatted drives do a good job recovering from power failures.  I am guessing that the sync errors may be due to the parity drive not recovering quite as well.  Does the fact that drive's journal activity is overlapping with parity checking create any data integrity risk (I think not, but hearing your thoughts would be assuring!)?

 

I am picking up my UPS this afternoon!  Our power has been very stable, but two outages in two days is too much.  We'll probably never have another power outage since I'll be ready for it!

 

One last question - if there is a power outage and I need to shutdown the unRAID server from the console WITHOUT being able to see the monitor, is there a command (or set of commands) that I can enter to bring down the array smoothly and power down the server.  (I notice there is an enhancement request to allow a UPS to shutdown the server, but until that is done, I'd like to know how to do it manually).  Thanks!

 

-Brian

Epilogue:

 

- No ill effects from 2 power outages and associated parity errors - no data corruption I've been able to find

- Installed UPS - will guarantee no power outages for the forseeable future

- Did some research and found the following commands will faithfully shutdown unraid from the console:

sync
stop
powerdown

 

Of course, you need to be logged on first, so to do a "blind" powerdown, I'd enter:

root      (to log on)
            (hit enter again to respond to password prompt)
sync
stop
powerdown

I'm considering a few additions of my own using the X10 subsystem.

 

Currently I have an X10 remote control environment.

I was considering a port of the daemons to unRaid.

 

With the proper equipment off the serial port, you would be able to send a remote control sequence to power off the machine.

(Powering on is a different story)

 

Another thing you could consider.. if you have the extra cash.

 

I purchased a viewsonic V210 airsync wireless display of eBay.

http://www.viewsonic.com/products/mobilewireless/wirelessmonitors/airsyncv210wirelessdisplay/index.htm

 

The cool thing about this model is it's like a PDA and has internet explorer on it.

Therefore you can connect to the webserver and with a few clicks, power down.

 

Then again, if you purchased a cheap used PDA with wireless (or a new fangled phone with wireless).

You can use those browsers to connect and power down.

 

 

In my particular case, I have a few ITX machines that babysit my network

(overlord and underlord).

 

Overlord has the X10 hook up, so I have it rigged to power on unraid with a wake on lan with one signal

and issue ssh commands to shutdown the machine on other signals.

 

I may add triggers to spin all drives up and spin all drives down. (just have not gotten around to it yet)

 

So a number of possibilities exist.

 

 

One more possible solution is with some server board combos you can redirect the bios screens to a serial port (supermicros are good with this).  I believe linux can be configured to use a serial port for console too. But this would require another machine to be active along with a terminal emulator for control  In my case Overlord/Underlord have plenty of serial ports for this usage.

It's probably not practical for the norm.

 

There is also KVM over IP, but this can be costly too.

 

Last but not least.

I'm sure it's possible to alter the CTRL-ALT-DEL sequence to issue a shutdown halt instead of a reboot.

 

This line would be changed.

# What to do at the "Three Finger Salute".

ca::ctrlaltdel:/sbin/shutdown -t5 -r now

 

 

I know you would need -h instead of -r.

However there would need to be other commands run before this.

 

 

I  bet it could be rigged with.

 

ca::ctrlaltdel:/bin/sync;/root/mdcmd stop;/bin/sleep 3;exec /sbin/poweroff

 

Perhaps an external script would be better.

I'll play a lil with it.

 

Just some ideas.