October 28, 201015 yr Up until about 24 hours ago my unRAID server was working normally. I had actually done a full parity check about 5 days ago which passed with no errors. Over the past few weeks, I had recently started to see a number of errors like the one I reported here: http://lime-technology.com/forum/index.php?topic=8335.0 but the array has otherwise been running fine, practically 24/7, for over a year now. The motherboard is a Gigabyte EP45-UD3P. This morning I noticed something peculiar: trying to generate a new torrent file for an ISO image on the server, then performing a recheck, would show missing pieces. Regenerating the file several times would produce different degrees of missing pieces (i.e. inconsistent results). After 6 tries it worked correctly. By this evening, however, when I got home from work, things seemed much worse. I isolated it to the unRAID server by using different workstations. In testing it became impossible to create any torrents from unRAID-based files that would work, and suddenly I began to find that torrents that were seeding (downloaded long ago and transferred to the unRAID server) would no longer recheck 100%. This occurs for many, many different files. Starting a parity check now produced tens of thousands of errors (44800+ in the first 0.1%, at which point I halted it). The server can run at least 4 passes of memtest without errors. 1. I redownloaded a sample video file to a clean computer via P2P from the original source. Running md5sum local to unRAID on this file gave me a different value than the clean file on the clean computer, indicating there is now a difference between the two that did not exist previously. Note this file was randomly chosen out of a 20TB collection so I don't think I "got lucky" and caught one of a select few bad files on my system. This file also existed on the server several months ago and should not have been altered since. 2. I copied the same file from unRAID to the clean computer. Playing it in a media player shows glitches, indicating the file is corrupted. Checking the file in uTorrent shows the same corruption when checked against , which would indicate that unRAID is behaving "consistently"... either its delivery of data over the network always gives the exact same corrupted version, or, more likely, the file is corrupted locally on the server and copying it to the clean computer. 3. In some cases I have *.par2 files in some picture directories. Running a par check indicates files are corrupted... but it seems like it's a "bit here and a byte there" (see screenshot). Ideally I'd run a par2 check locally but I'm pretty sure its all local to the server now. If I use unMenu to run a "Parity Verify" (without correction) it will still show 1000's of errors. Seven drives are on a Supermicro AOC-SASLP-MV8. Files are scattered throughout and show problems when on drives on the Supermicro or on the motherboard's Intel controller. The array has had maybe 3-4GB of file write activity over the past two days. I'm not sure what to do now. I will consider replacing the motherboard ASAP, but I'm very confident it's not memory or CPU given the consistency of the errors and overall stability otherwise (the system is not crashing and repeated md5sums on the same files give the same results). Anyway, I'm running reiserfsck --check against all the partitions, but so far they seem to be coming out OK. This will take some time for 14 2TB drives. What do you think is the current state? Should I do a parity rebuild, run a full check, or...?? If hardware based corruption has occurred, is there no chance of getting the data back to a consistent state? (Most of the stuff is media that can probably survive low density bit errors, but obviously a few files seems to be in bad shape- the sample I tried in #1 was glitchy). Side note: I have three empty 2TB hard drives (currently members of the array) to work with.
October 28, 201015 yr Author Update: the partitions are not passing reiserfsck in all cases. Most look like this: ########### reiserfsck --check started at Thu Oct 28 01:24:00 2010 ########### Replaying journal: Trans replayed: mountid 64, transid 36858, desc 198, len 1, commit 200, next trans offset 183 Trans replayed: mountid 64, transid 36859, desc 201, len 1, commit 203, next trans offset 186 Replaying journal: Done. Reiserfs journal '/dev/sdj1' in blocks [18..8211]: 2 transactions replayed Checking internal tree.. \/ 14 (of 19-/ 82 (of 129|/117 (of 170/bad_indirect_item: block 405995677: The item (8210 8224 0x2673c001 IND (1), len 4048, location 48 entry count 0, fsck need 0, format new) has the bad pointer (524) to the block (406156711), which is in tree already / 17 (of 19-/158 (of 170\/ 91 (of 170-block 122321549: The level of the node (16458) is not correct, (1) expected the problem in the internal node occured (122321549), whole subtree is skipped finished Comparing bitmaps..vpf-10640: The on-disk and the correct bitmaps differs. Bad nodes were found, Semantic pass skipped 1 found corruptions can be fixed only when running with --rebuild-tree ########### reiserfsck finished at Thu Oct 28 01:36:23 2010 ########### One (so far) looks like: ########### reiserfsck --check started at Thu Oct 28 01:21:29 2010 ########### Replaying journal: Trans replayed: mountid 115, transid 35177, desc 4134, len 1, commit 4136, next trans offset 4119 Trans replayed: mountid 115, transid 35178, desc 4137, len 1, commit 4139, next trans offset 4122 Replaying journal: Done. Reiserfs journal '/dev/sdi1' in blocks [18..8211]: 2 transactions replayed Checking internal tree.. finished Comparing bitmaps..finished Checking Semantic tree: ... rt Travels S4Ep01 - Edinburgh & Scotland 720p HDTV Mpeg2 DD5.1.tsufile.c 391 are_file_items_correct are_file_items_correct: Position (offset == 15552513) in the middle ofthe file [30968 30989] was not found. Aborted
October 28, 201015 yr Update: the partitions are not passing reiserfsck in all cases. Most look like this: ########### reiserfsck --check started at Thu Oct 28 01:24:00 2010 ########### Replaying journal: Trans replayed: mountid 64, transid 36858, desc 198, len 1, commit 200, next trans offset 183 Trans replayed: mountid 64, transid 36859, desc 201, len 1, commit 203, next trans offset 186 Replaying journal: Done. Reiserfs journal '/dev/sdj1' in blocks [18..8211]: 2 transactions replayed Checking internal tree.. \/ 14 (of 19-/ 82 (of 129|/117 (of 170/bad_indirect_item: block 405995677: The item (8210 8224 0x2673c001 IND (1), len 4048, location 48 entry count 0, fsck need 0, format new) has the bad pointer (524) to the block (406156711), which is in tree already / 17 (of 19-/158 (of 170\/ 91 (of 170-block 122321549: The level of the node (16458) is not correct, (1) expected the problem in the internal node occured (122321549), whole subtree is skipped finished Comparing bitmaps..vpf-10640: The on-disk and the correct bitmaps differs. Bad nodes were found, Semantic pass skipped 1 found corruptions can be fixed only when running with --rebuild-tree ########### reiserfsck finished at Thu Oct 28 01:36:23 2010 ########### One (so far) looks like: ########### reiserfsck --check started at Thu Oct 28 01:21:29 2010 ########### Replaying journal: Trans replayed: mountid 115, transid 35177, desc 4134, len 1, commit 4136, next trans offset 4119 Trans replayed: mountid 115, transid 35178, desc 4137, len 1, commit 4139, next trans offset 4122 Replaying journal: Done. Reiserfs journal '/dev/sdi1' in blocks [18..8211]: 2 transactions replayed Checking internal tree.. finished Comparing bitmaps..finished Checking Semantic tree: ... rt Travels S4Ep01 - Edinburgh & Scotland 720p HDTV Mpeg2 DD5.1.tsufile.c 391 are_file_items_correct are_file_items_correct: Position (offset == 15552513) in the middle ofthe file [30968 30989] was not found. Aborted First... verify the memory test is still good. If it is not, the reiserfsck results are suspect. I suspect the data on the disks may be fine, but the memory/cpu/motherboard not. Odds of it being multiple disks at the same time is slim unless there is a common cause.
October 28, 201015 yr Author Thanks Joe. You're always above and beyond the call of duty... I'm taking advantage of the deal at NewEgg... actually got a SUPERMICRO MBD-C2SEE-O for $21.64 shipped All the reiserfscks are now complete (wish I could open more than 8 PuTTY sessions ). FWIW (keeping in mind these may be suspect) excluding parity: 1. The 3 empty drives all checked OK (of course... there's no data and the checks were complete so if it's RAM then it probably gets away with finishing). 2. 1 drive wants "--fix-fixable" 3. 3 drives that are mostly full check OK 4. 6 drives want "--rebuild-tree" (one drive I don't care as much about as it is reserved for network backups - when all this is resolved I will almost certainly just delete the contents and regenerate backups) 5. 1 drive reiserfsck aborts (as mentioned in my second post) I'm skeptical that RAM is the cause (I guess it's not what I'm used to seeing when RAM goes bad... but I'm keeping an open mind) for these reasons: I can repeatedly copy a file to a workstation off the server and it will md5sum identically local to the unRAID server and the workstation. (which also kinda rules out a NIC issue) The system is not crashing The above reiserfsck behavior is repeatedly reproducible for each drive (I almost wish it wasn't) - i.e. reiserfsck give the same specific results every time for the same specific drives md5sums run locally are the same every time torrent re-check scans show the same pieces broken every time when checked against the array, or a file copied from the array to a local workstation Several of the drives were basically spun down most of the time and were never written to so I'm hopeful the data is still OK. However, of the 3 drives with data that report OK, one of them is the one that contains the media file that I tested as corrupt (at least when copied over to a workstation). I'm a little more suspicious of the combination of the Gigabyte EP45-UD3P's PCI-E x16 slots + Supermicro AOC-SASLP-MV8 + mvsas driver issue that seems to occur on multi-core/SMP machines. I'd had the controller board in the PCI-E x16 first slot and switched it to the other slot a few days ago when I was trying to resolve the kernel crashes, though it parity checked 100% clean after that point. This may have been a contributing factor. Anyway thanks again Joe... even your short response prevented me from going too far (e.g. running reiserfsck in what could be a bad attempt to "fix" something that isn't broken). I'm cautiously optimistic that the data is OK in most cases (many of these drives were probably spun down 95-99.9% of the time over the past 5 days when the array checked fine and all but a couple would have had any files written to them in that period) but I'm prepared to accept the worst. I guess now that I've had some sleep and some time, my next step will be to take all the drives offline and put the unRAID server into perpetual memtest and wait for the new mobo and RAM and mount drives one by one onto the new system and go from there... but I'm certainly open to anyone's further input.
October 29, 201015 yr Author 5 passes of memtest over 12 hours and no errors... my Supermicro board arrives today; will work on it tonight.
October 30, 201015 yr Author The new mobo is in place and magically my data appears to be OK. I'm running reiserfsck on the data disk partitions and so far so good. Damage: The first 0.1% of the parity drive had about 45000 errors, because I had corrupted it when I panicked initially (see OP... before I discovered unMenu has a "Verify Parity" function). Also, the last file written to the array, an ISO, does appear to have some bad data but I can reconstruct it from the original DVD. Fortunately, other than this file I had done no write activity to the array since the last (good) parity check five days prior. I'd actually ordered another Supermicro card and cables to try as well if this hadn't worked but I guess those will be what I use to add more disks when I need to. Since memory checked OK on the old board and CPU is stable, I'm going to presume there was a hardware failure on the PCI-E bus of some kind on the original mobo because of the consistency of the errors I was getting. It does bother me a little that unRAID gave nary an error as I was writing a corrupted file to one of its disks, but perhaps the kernel crashes I was starting to see were an early sign. If there's any takeaway I have from this, it's that if something starts going wrong, stop everything and don't try to fix it until you're 200% certain what your problem is.
October 30, 201015 yr Congratulations, and it's great to get such an inexpensive fix (neglecting the extra SASLP).
Archived
This topic is now archived and is closed to further replies.