February 13, 200917 yr Hello I have done a parity check and it returned with 117 errors: Feb 7 09:21:47 UNRAID kernel: ReiserFS: warning: is_tree_node: node level 55551 does not match to the expected one 1 Feb 7 09:21:47 UNRAID kernel: ReiserFS: md3: warning: vs-5150: search_by_key: invalid format found in block 5505202. Fsck? Feb 7 09:21:47 UNRAID kernel: ReiserFS: md3: warning: vs-13070: reiserfs_read_locked_inode: i/o failure occurred trying to find stat data of [141 146 0x0 SD] Feb 7 09:21:47 UNRAID kernel: ReiserFS: warning: is_tree_node: node level 55551 does not match to the expected one 1 Feb 7 09:21:47 UNRAID kernel: ReiserFS: md3: warning: vs-5150: search_by_key: invalid format found in block 5505202. Fsck? Feb 7 09:21:47 UNRAID kernel: ReiserFS: md3: warning: vs-13070: reiserfs_read_locked_inode: i/o failure occurred trying to find stat data of [141 144 0x0 SD] All error messages are the same as above and for the same block 5505202. Wich drive is broken? Disk 3??? Regards Shootking
February 14, 200917 yr Yes, you need to run reiserfsck on Disk 3. Please see the Check Disk Filesystems page. You will use md3 wherever the examples use md1. Follow the instructions very carefully. Once all fixed, run another parity check, and don't be surprised to see more parity errors reported. But this time it should be correctly fixing them all.
February 14, 200917 yr Author If I understand this correct, this is a drive that may be broken physically. What I dont understand is how unRaid is protecting me from this hardware error. IS the software able to correct these errors or is the parity drive calculationg parity from the broken/false data? I am running reiserfsck --rebuild-tree as the reiserfsck suggested. The cehck reported: Replaying journal.. Reiserfs journal '/dev/md3' in blocks [18..8211]: 0 transactions replayed Checking internal tree../ 2 (of 4)/ 26 (of 153)/ 57 (of 85)block 5505202: The level of the node (55551) is not correct, (1) expected the problem in the internal node occuredfinished Comparing bitmaps..vpf-10640: The on-disk and the correct bitmaps differs. Bad nodes were found, Semantic pass skipped 1 found corruptions can be fixed only when running with --rebuild-tree The same block mentioned in te syslog is reported here too. /Shootking
February 14, 200917 yr Author Ooops, when I tried to copy the text from my Putty window I used CTRL-C wich made the reiserfsck --rebuild-tree to stop... I hope that I havent screwed things up now...Running the check again now. /Shootking
February 14, 200917 yr Author ########### reiserfsck --rebuild-tree started at Sat Feb 14 10:12:05 2009 ########### Pass 0: ####### Pass 0 ####### Loading on-disk bitmap .. ok, 96681532 blocks marked used Skipping 11191 blocks (super block, journal, bitmaps) 96670341 blocks will be read 0%....20%....40%. left 03344 directory entries were hashed with "r5" hash. "r5" hash is selected Flushing..finished Read blocks (but not data blocks) 96670341 Leaves among those 96037 Objectids found 3350 Pass 1 (will try to insert 96037 leaves): ####### Pass 1 ####### Looking for allocable blocks .. finished 0%....20%....40%....60%....80%....100% left 0, 310 /sec Flushing..finished 96037 leaves read 95975 inserted - pointers in indirect items pointing to metadata 1 (zeroed) 62 not inserted non-unique pointers in indirect items (zeroed) 636 ####### Pass 2 ####### Pass 2: 0%....20%....40%....60%....80%....100% left 0, 41 /sec Flushing..finished Leaves inserted item by item 62 Pass 3 (semantic): ####### Pass 3 ######### /dvd/T2 - directors cut/mymovies-front.jpgvpf-10680: The file [141 144] has the wrong block count in the StatData (160) - corrected to (16) /folder.jpgvpf-10680: The file [141 142] has the wrong block count in the StatData (104) - corrected to (0) /Jasper - Sova i T??lt/Thumbs.dbvpf-10680: The file [2310 3742] has the wrong block count in the StatData (16) - corrected to ( /Fantastic 4 and the silver surf Flushing..finished Files found: 3072 Directories found: 274 Pass 3a (looking for lost dir/files): ####### Pass 3a (lost+found pass) ######### Looking for lost directories: Flushing..finished1, 294 /sec Pass 4 - finished done 0, 0 /sec Flushing..finished Syncing..finished ########### reiserfsck finished at Sat Feb 14 12:02:47 2009 ########### It seems that three files are corrupt: /dvd/T2 - directors cut/mymovies-front.jpg /folder.jpg /Jasper - Sova i T??lt/Thumbs.db /Fantastic 4 and the silver surf There was no filename on the last entry. The feeling i get is that I should replace this drive Should I also run reiserfsch on my other drives as some kind of helath check? Should I do this regulary? How are you other guys doing in order to keep your drives healthy? Regards
February 14, 200917 yr Author Perity check after reiserfsck command resulted in no errors: (Last checked on 2/14/2009 6:02:50 PM, finding 0 errors.) What should I do? Is the disk bad or was it only the file system (from an unexpected shutdown for example) Regards Shootking
February 14, 200917 yr Here are a few snippets that might help to explain the situation ... 1. When you run a parity check, unRAID will FIX any parity sync errors it finds (by updating the parity disk). That means if you run it and get 10 sync errors, and then run it again and get 0, it doesn't mean the original 10 were a fluke. It just means that unRAID has already made the change in parity. 2. The data disks are formatted with the ReiserFS file system. Despite the fact that the author killed his wife and is now rotting in jail (yes, that is true), it is a very good file system. In particular it gets very high marks in data integrity even after a dirty shutdown. If you had just a regular Linux install and weren't using unRAID, you'd be depending on these disks to not get corrupted, much the way you depend on NTFS to keep your disks accurate. Again, this is a very good FileSystem in very widespread use and the chances that it is going to get corrupted EXTREMELY small. 3. unRAID's parity disk (and I hope this doesn't come across wrong), is much less widely used and is NOT using a traditional file system. It is only updated when it is "watching" the other disks, and in cases of power failure or potentially other nasty events it can lose track of some of the updates to the data disks. Don't get me wrong, the parity disk does a GREAT job of staying in sync with the data disks, but if there is an out-of-sync condition, you have to suspect that the parity disk is the likely culprit and not the data disk. 4. In cases of power outage, in particular, sync errors are common, even normal. Sync errors typically happen in the "housekeeping" part of the disk, and it is common to see a flurry of sync errors soon after the parity check starts, and few if any for the remained of the parity check. 5. To summarize unRAID's view of parity, it has no operational significance while the array is whole. If its wrong (i.e., not in sync with the data), it fixes it during a parity check. But otherwise, unRAID doesn't really give a crap. Some people install their arrays and NEVER do a parity check. Sometimes, when told months later they should be doing them, they find thousands of parity errors, and also find that they get even more when they run it a second time, third time, etc. This is a symptom of motherboard compatibility. But these users ran perfectly fine with an illusion of redundancy for months, and likely could have done so for many more months or even years. BAD PARITY IS COMPLETELY INVISIBLE UNLESS YOU DO PARITY CHECKS! 6. There are only a few times when unRAID really depends on parity. It is during these times, and these times only, that I get concerned about parity errors. Important clarification - parity errors that are found and corrected during parity checks are not parity errors anymore. The parity errors that I am concerned about are the ones that HAVEN'T been found, and therefore represent an UNKNOWN/unhandled synchronization issue. a. During drive rebuilds (failed disk or upsizing disk). EACH AND EVERY SECTOR ON PARITY HAD BETTER BE RIGHT, otherwise the rebuild will have an error. (This is not really 100% true, there are areas of the housekeeping area (that might be caused by having to reboot after a system crash caused by the failing disk) that can get updated and have no consequential impact of actual data. But in general the parity disk has to be right. b. Bad sectors. During normal operation, it is not that unusual for a bad sector to occur. In these cases, unRAID will immediately query all of the drives in the array (+ parity) and decide on what that data should have been at that sector. It will then update that data correcting the bad sector. This is only one sector that must be right on parity, not the whole disk. But it is another opportunity for data to get corrupted if parity is random gobblty gook. ------------------ Conclusion, if you didn't do a drive rebuild or have a bad sector (error column in Web GUI will show these), the chances that you actually had any data corruption is very very low. If you had a hard shutdown or experimented with some Linux commands to mount / unmount or write to the devices directly you could expect that you would have some parity sync issues, but if you didn't it is a mystery. I would recommend running parity checks often (1-2 times a week for the next month or so) so that you are confident that the parity is being maintained correctly. If you continue to see some parity errors, post back and there will be ideas about things you should check. It could point to incompatibility - but my experience is that incompatible motherboards spit out 1000s of sync errors each and every time a parity check it run. Hope this helps ...
February 15, 200917 yr If I understand this correct, this is a drive that may be broken physically. What I dont understand is how unRaid is protecting me from this hardware error. IS the software able to correct these errors or is the parity drive calculationg parity from the broken/false data? No, there is absolutely nothing wrong with the drive. This is the same as pulling the cord out on a Windows machine while it was writing to the drive, and after rebooting, needing to run a scandisk (essentially the same as a reiserfsck) to fix the file system, and possibly finding one or more corrupted files. This is an extremely rare occurrence with the Reiser file system. Because of the corruption, this is most likely from either a crash or sudden power outage or major electrical spike, while something was being saved to an unRAID disk. The parity drive would also have a little corruption at the corresponding blocks where the writes were occurring, both to the fles involved as well as to the file system structures. The reiserfsck tool fixes the file system corruption, then a parity check fixes the out-of-sync parity. It is unfortunate that you also had files corrupted, something that is even more rare with the Reiser file system. Generally, reiserfsck, like scandisk, is only run when there is an indication of file system trouble. You could run it once in awhile if you wanted, but I don't think most do. You should only run the simple version (reiserfsck /dev/md1), and ONLY use --rebuild-tree and other similar options when reiserfsck itself instructs you to.
February 15, 200917 yr Author Thanks bjp999 and RobJ for your explanation. Now I have a much better understanding for my system. I will keep an eye for more parity check errors that could indicate hardware failure. Regards Shootking
Archived
This topic is now archived and is closed to further replies.