June 27, 201214 yr Hello Everyone, I have been running unraid for a short time and with success, but yesterday I attempted to add two additional drives to my array. I started the preclear scripts and came back 3 hours later and they had failed, and the server was not responding. I tried rebooting and trying again, with the same issue, etc. The syslog shows the following: Jun 26 23:32:41 UNRAID kernel: REISERFS error (device sdm1): vs-5150 search_by_key: invalid format found in block 90669730. Fsck? (Errors) Jun 26 23:32:41 UNRAID kernel: REISERFS error (device sdm1): vs-13070 reiserfs_read_locked_inode: i/o failure occurred trying to find stat data of [212955 449328 0x0 SD] (Errors) Jun 26 23:32:41 UNRAID kernel: REISERFS warning: reiserfs-5090 is_tree_node: node level 22397 does not match to the expected one 1 (Minor Issues) Jun 26 23:32:41 UNRAID kernel: REISERFS error (device sdm1): vs-5150 search_by_key: invalid format found in block 90610486. Fsck? (Errors) Jun 26 23:32:41 UNRAID kernel: REISERFS error (device sdm1): vs-13070 reiserfs_read_locked_inode: i/o failure occurred trying to find stat data of [212955 448479 0x0 SD] (Errors) Jun 26 23:32:41 UNRAID kernel: REISERFS warning: reiserfs-5090 is_tree_node: node level 50090 does not match to the expected one 1 (Minor Issues) Jun 26 23:32:41 UNRAID kernel: REISERFS error (device sdm1): vs-5150 search_by_key: invalid format found in block 90513399. Fsck? (Errors) Jun 26 23:32:41 UNRAID kernel: REISERFS error (device sdm1): vs-13070 reiserfs_read_locked_inode: i/o failure occurred trying to find stat data of [212955 447100 0x0 SD] (Errors) my drive configuration looks like this: parity sda Seagate 3T disk1 sdb Seagate 3T disk2 sdc Seagate 3T disk3 sdd Seagate 3T disk4 sde Seagate 3T disk5 sdf Seagate 2T disk6 sdg Seagate 1.5T disk7 sdh Seagate 1.5T disk8 sdl WD 1T flash sdi Unknown 16G new sdk Seagate 3T new sdj Seagate 3T cache sdm Seagate 500G I found this topic "http://lime-technology.com/forum/index.php?topic=20984.0" which pointed me to here: "http://lime-technology.com/wiki/index.php/Check_Disk_Filesystems" but im not sure if that is the correct next thing I should do. I am very new to linux and I feel in over my head at this point, any guidance would be much appreciated! Thank You
June 27, 201214 yr Folllow the instructions given in Check File systems. It needs to run on the cache drive (sdm). umount /dev/sdm reiserfsck --check /dev/sdm1 Do not forget the 1 (one).
June 28, 201214 yr Author Hello dgaschk, and everyone, I copied as much output as I could from the reiserfsck command to the block (90397625), which is in tree already bad_indirect_item: block 90243169: The item (212955 445036 0x1 IND (1), len 244, location 1792 entry count 0, fsck need 0, format new) has the bad pointer (53) to the block (90397626), which is in tree already bad_indirect_item: block 90243169: The item (212955 445036 0x1 IND (1), len 244, location 1792 entry count 0, fsck need 0, format new) has the bad pointer (54) to the block (90397627), which is in tree already bad_indirect_item: block 90243169: The item (212955 445036 0x1 IND (1), len 244, location 1792 entry count 0, fsck need 0, format new) has the bad pointer (55) to the block (90397628), which is in tree already bad_indirect_item: block 90243169: The item (212955 445036 0x1 IND (1), len 244, location 1792 entry count 0, fsck need 0, format new) has the bad pointer (56) to the block (90397629), which is in tree already bad_indirect_item: block 90243169: The item (212955 445036 0x1 IND (1), len 244, location 1792 entry count 0, fsck need 0, format new) has the bad pointer (57) to the block (90397630), which is in tree already bad_indirect_item: block 90243169: The item (212955 445036 0x1 IND (1), len 244, location 1792 entry count 0, fsck need 0, format new) has the bad pointer (58) to the block (90397631), which is in tree already bad_indirect_item: block 90243169: The item (212955 445036 0x1 IND (1), len 244, location 1792 entry count 0, fsck need 0, format new) has the bad pointer (59) to the block (90397632), which is in tree already bad_indirect_item: block 90243169: The item (212955 445036 0x1 IND (1), len 244, location 1792 entry count 0, fsck need 0, format new) has the bad pointer (60) to the block (90397633), which is in tree already bad_indirect_item: block 90243169: The item (212955 445038 0x1 IND (1), len 244, location 1216 entry count 0, fsck need 0, format new) has the bad pointer (0) to the block (90397634), which is in tree already bad_indirect_item: block 90243169: The item (212955 445038 0x1 IND (1), len 244, location 1216 entry count 0, fsck need 0, format new) has the bad pointer (1) to the block (90397635), which is in tree already bad_indirect_item: block 90243169: The item (212955 445038 0x1 IND (1), len 244, location 1216 entry count 0, fsck need 0, format new) has the bad pointer (2) to the block (90397636), which is in tree already bad_indirect_item: block 90243169: The item (212955 445038 0x1 IND (1), len 244, location 1216 entry count 0, fsck need 0, format new) has the bad pointer (3) to the block (90397637), which is in tree already bad_indirect_item: block 90243169: The item (212955 445038 0x1 IND (1), len 244, location 1216 entry count 0, fsck need 0, format new) has the bad pointer (4) to the block (90397638), which is in tree already bad_indirect_item: block 90243169: The item (212955 445038 0x1 IND (1), len 244, location 1216 entry count 0, fsck need 0, format new) has the bad pointer (5) to the block (90397639), which is in tree already bad_indirect_item: block 90243169: The item (212955 445038 0x1 IND (1), len 244, location 1216 entry count 0, fsck need 0, format new) has the bad pointer (6) to the block (90397640), which is in tree already bad_indirect_item: block 90243169: The item (212955 445038 0x1 IND (1), len 244, location 1216 entry count 0, fsck need 0, format new) has the bad pointer (7) to the block (90397641), which is in tree already bad_indirect_item: block 90243169: The item (212955 445038 0x1 IND (1), len 244, location 1216 entry count 0, fsck need 0, format new) has the bad pointer ( to the block (90397642), which is in tree already bad_indirect_item: block 90243169: The item (212955 445038 0x1 IND (1), len 244, location 1216 entry count 0, fsck need 0, format new) has the bad pointer (9) to the block (90397643), which is in tree already bad_indirect_item: block 90243169: The item (212955 445038 0x1 IND (1), len 244, location 1216 entry count 0, fsck need 0, format new) has the bad pointer (10) to the block (90397644), which is in tree already bad_indirect_item: block 90243169: The item (212955 445038 0x1 IND (1), len 244, location 1216 entry count 0, fsck need 0, format new) has the bad pointer (11) to the block (90397645), which is in tree already bad_indirect_item: block 90243169: The item (212955 445038 0x1 IND (1), len 244, location 1216 entry count 0, fsck need 0, format new) has the bad pointer (12) to the block (90397646), which is in tree already bad_indirect_item: block 90243169: The item (212955 445038 0x1 IND (1), len 244, location 1216 entry count 0, fsck need 0, format new) has the bad pointer (13) to the block (90397647), which is in tree already bad_indirect_item: block 90243169: The item (212955 445038 0x1 IND (1), len 244, location 1216 entry count 0, fsck need 0, format new) has the bad pointer (14) to the block (90397648), which is in tree already bad_indirect_item: block 90243169: The item (212955 445038 0x1 IND (1), len 244, location 1216 entry count 0, fsck need 0, format new) has the bad pointer (15) to the block (90397649), which is in tree already bad_indirect_item: block 90243169: The item (212955 445038 0x1 IND (1), len 244, location 1216 entry count 0, fsck need 0, format new) has the bad pointer (16) to the block (90397650), which is in tree already bad_indirect_item: block 90243169: The item (212955 445038 0x1 IND (1), len 244, location 1216 entry count 0, fsck need 0, format new) has the bad pointer (17) to the block (90397651), which is in tree already bad_indirect_item: block 90243169: The item (212955 445038 0x1 IND (1), len 244, location 1216 entry count 0, fsck need 0, format new) has the bad pointer (18) to the block (90397652), which is in tree already bad_indirect_item: block 90243169: The item (212955 445038 0x1 IND (1), len 244, location 1216 entry count 0, fsck need 0, format new) has the bad pointer (19) to the block (90397653), which is in tree already bad_indirect_item: block 90243169: The item (212955 445038 0x1 IND (1), len 244, location 1216 entry count 0, fsck need 0, format new) has the bad pointer (20) to the block (90397654), which is in tree already bad_indirect_item: block 90243169: The item (212955 445038 0x1 IND (1), len 244, location 1216 entry count 0, fsck need 0, format new) has the bad pointer (21) to the block (90397655), which is in tree already bad_indirect_item: block 90243169: The item (212955 445038 0x1 IND (1), len 244, location 1216 entry count 0, fsck need 0, format new) has the bad pointer (22) to the block (90397656), which is in tree already bad_indirect_item: block 90243169: The item (212955 445038 0x1 IND (1), len 244, location 1216 entry count 0, fsck need 0, format new) has the bad pointer (23) to the block (90397657), which is in tree already bad_indirect_item: block 90243169: The item (212955 445038 0x1 IND (1), len 244, location 1216 entry count 0, fsck need 0, format new) has the bad pointer (24) to the block (90397658), which is in tree already bad_indirect_item: block 90243169: The item (212955 445038 0x1 IND (1), len 244, location 1216 entry count 0, fsck need 0, format new) has the bad pointer (25) to the block (90397659), which is in tree already bad_indirect_item: block 90243169: The item (212955 445038 0x1 IND (1), len 244, location 1216 entry count 0, fsck need 0, format new) has the bad pointer (26) to the block (90397660), which is in tree already bad_indirect_item: block 90243169: The item (212955 445038 0x1 IND (1), len 244, location 1216 entry count 0, fsck need 0, format new) has the bad pointer (27) to the block (90397661), which is in tree already bad_indirect_item: block 90243169: The item (212955 445038 0x1 IND (1), len 244, location 1216 entry count 0, fsck need 0, format new) has the bad pointer (28) to the block (90397662), which is in tree already bad_indirect_item: block 90243169: The item (212955 445038 0x1 IND (1), len 244, location 1216 entry count 0, fsck need 0, format new) has the bad pointer (29) to the block (90397663), which is in tree already bad_indirect_item: block 90243169: The item (212955 445038 0x1 IND (1), len 244, location 1216 entry count 0, fsck need 0, format new) has the bad pointer (30) to the block (90397664), which is in tree already bad_indirect_item: block 90243169: The item (212955 445038 0x1 IND (1), len 244, location 1216 entry count 0, fsck need 0, format new) has the bad pointer (31) to the block (90397665), which is in tree already bad_indirect_item: block 90243169: The item (212955 445038 0x1 IND (1), len 244, location 1216 entry count 0, fsck need 0, format new) has the bad pointer (32) to the block (90397666), which is in tree already bad_indirect_item: block 90243169: The item (212955 445038 0x1 IND (1), len 244, location 1216 entry count 0, fsck need 0, format new) has the bad pointer (33) to the block (90397667), which is in tree already bad_indirect_item: block 90243169: The item (212955 445038 0x1 IND (1), len 244, location 1216 entry count 0, fsck need 0, format new) has the bad pointer (34) to the block (90397668), which is in tree already bad_indirect_item: block 90243169: The item (212955 445038 0x1 IND (1), len 244, location 1216 entry count 0, fsck need 0, format new) has the bad pointer (35) to the block (90397669), which is in tree already bad_indirect_item: block 90243169: The item (212955 445038 0x1 IND (1), len 244, location 1216 entry count 0, fsck need 0, format new) has the bad pointer (36) to the block (90397670), which is in tree already bad_indirect_item: block 90243169: The item (212955 445038 0x1 IND (1), len 244, location 1216 entry count 0, fsck need 0, format new) has the bad pointer (37) to the block (90397671), which is in tree already bad_indirect_item: block 90243169: The item (212955 445038 0x1 IND (1), len 244, location 1216 entry count 0, fsck need 0, format new) has the bad pointer (38) to the block (90397672), which is in tree already bad_indirect_item: block 90243169: The item (212955 445038 0x1 IND (1), len 244, location 1216 entry count 0, fsck need 0, format new) has the bad pointer (39) to the block (90397673), which is in tree already bad_indirect_item: block 90243169: The item (212955 445038 0x1 IND (1), len 244, location 1216 entry count 0, fsck need 0, format new) has the bad pointer (40) to the block (90397674), which is in tree already bad_indirect_item: block 90243169: The item (212955 445038 0x1 IND (1), len 244, location 1216 entry count 0, fsck need 0, format new) has the bad pointer (41) to the block (90397675), which is in tree already bad_indirect_item: block 90243169: The item (212955 445038 0x1 IND (1), len 244, location 1216 entry count 0, fsck need 0, format new) has the bad pointer (42) to the block (90397676), which is in tree already bad_indirect_item: block 90243169: The item (212955 445038 0x1 IND (1), len 244, location 1216 entry count 0, fsck need 0, format new) has the bad pointer (43) to the block (90397677), which is in tree already bad_indirect_item: block 90243169: The item (212955 445038 0x1 IND (1), len 244, location 1216 entry count 0, fsck need 0, format new) has the bad pointer (44) to the block (90397678), which is in tree already bad_indirect_item: block 90243169: The item (212955 445038 0x1 IND (1), len 244, location 1216 entry count 0, fsck need 0, format new) has the bad pointer (45) to the block (90397679), which is in tree already bad_indirect_item: block 90243169: The item (212955 445038 0x1 IND (1), len 244, location 1216 entry count 0, fsck need 0, format new) has the bad pointer (46) to the block (90397680), which is in tree already bad_indirect_item: block 90243169: The item (212955 445038 0x1 IND (1), len 244, location 1216 entry count 0, fsck need 0, format new) has the bad pointer (47) to the block (90397681), which is in tree already bad_indirect_item: block 90243169: The item (212955 445038 0x1 IND (1), len 244, location 1216 entry count 0, fsck need 0, format new) has the bad pointer (48) to the block (90397682), which is in tree already bad_indirect_item: block 90243169: The item (212955 445038 0x1 IND (1), len 244, location 1216 entry count 0, fsck need 0, format new) has the bad pointer (49) to the block (90397683), which is in tree already bad_indirect_item: block 90243169: The item (212955 445038 0x1 IND (1), len 244, location 1216 entry count 0, fsck need 0, format new) has the bad pointer (50) to the block (90397684), which is in tree already bad_indirect_item: block 90243169: The item (212955 445038 0x1 IND (1), len 244, location 1216 entry count 0, fsck need 0, format new) has the bad pointer (51) to the block (90397685), which is in tree already bad_indirect_item: block 90243169: The item (212955 445038 0x1 IND (1), len 244, location 1216 entry count 0, fsck need 0, format new) has the bad pointer (52) to the block (90397686), which is in tree already bad_indirect_item: block 90243169: The item (212955 445038 0x1 IND (1), len 244, location 1216 entry count 0, fsck need 0, format new) has the bad pointer (53) to the block (90397687), which is in tree already bad_indirect_item: block 90243169: The item (212955 445038 0x1 IND (1), len 244, location 1216 entry count 0, fsck need 0, format new) has the bad pointer (54) to the block (90397688), which is in tree already bad_indirect_item: block 90243169: The item (212955 445038 0x1 IND (1), len 244, location 1216 entry count 0, fsck need 0, format new) has the bad pointer (55) to the block (90397689), which is in tree already bad_indirect_item: block 90243169: The item (212955 445038 0x1 IND (1), len 244, location 1216 entry count 0, fsck need 0, format new) has the bad pointer (56) to the block (90397690), which is in tree already bad_indirect_item: block 90243169: The item (212955 445038 0x1 IND (1), len 244, location 1216 entry count 0, fsck need 0, format new) has the bad pointer (57) to the block (90397691), which is in tree already bad_indirect_item: block 90243169: The item (212955 445038 0x1 IND (1), len 244, location 1216 entry count 0, fsck need 0, format new) has the bad pointer (58) to the block (90397692), which is in tree already bad_indirect_item: block 90243169: The item (212955 445038 0x1 IND (1), len 244, location 1216 entry count 0, fsck need 0, format new) has the bad pointer (59) to the block (90397693), which is in tree already bad_indirect_item: block 90243169: The item (212955 445038 0x1 IND (1), len 244, location 1216 entry count 0, fsck need 0, format new) has the bad pointer (60) to the block (90397694), which is in tree already / 43 (of 169-block 90378440: The level of the node (38112) is not correct, (1) expected the problem in the internal node occured (90378440), / 37 (of 166\/ 1 (of 168|bad_path: The left delimiting key [212955 446719 0x0 SD (0)] of the node (90488261) must be equal to the first element's key [212955 446694 0x19001 IND (1)] within the node. / 2 (of 168/block 90481804: The level of the node (22543) is not correct, (1) expected the problem in the internal node occured (90481804), / 38 (of 166-/ 2 (of 168|block 90640934: The level of the node (53489) is not correct, (1) expected the problem in the internal node occured (90640934),finished Comparing bitmaps..vpf-10640: The on-disk and the correct bitmaps differs. Bad nodes were found, Semantic pass skipped 26 found corruptions can be fixed only when running with --rebuild-tree ########### reiserfsck finished at Wed Jun 27 20:42:30 2012 ########### I assume the beef of what I need to know is: Bad nodes were found, Semantic pass skipped 26 found corruptions can be fixed only when running with --rebuild-tree Should I run the command it suggests? which I assume would be: reiserfsck --rebuild-tree /dev/sdm1 Also, does this indicate a failing drive? Because if needed I can just replace it. Thank you
June 28, 201214 yr Author Thank you for helping me dgaschk! I have run the command reiserfsck --rebuild-tree /dev/sdm1 and am waiting to see if it repairs things, I have one last question. Would cold powering down Unraid cause these types of problems? I am guilty of doing that once or twice, is this something that just happens, or was it most likely caused by my actions? Thank you
June 28, 201214 yr Author I re-ran the check, and got the following: No corruptions found There are on the filesystem: Leaves 23770 Internal nodes 166 Directories 158 Other files 11262 Data block pointers 22798563 (13791 of them are zero) Safe links 0 ########### reiserfsck finished at Wed Jun 27 21:30:35 2012 ########### A HUGE thank you to dgaschk who really helped me out, thanks man!
June 28, 201214 yr Author Well, I thought we had fixed my problem, but I started up the preclear scripts, and started checking out the file system, and then the unraid webpage, and samba shares stopped responding. Now this is showing up in my log: Jun 27 21:43:19 UNRAID kernel: ata12.00: failed command: READ FPDMA QUEUED (Minor Issues) Jun 27 21:43:19 UNRAID kernel: ata12.00: cmd 60/00:00:00:19:29/01:00:04:00:00/40 tag 1 ncq 131072 in (Drive related) Jun 27 21:43:19 UNRAID kernel: res 01/04:00:00:1a:29/00:00:04:00:00/40 Emask 0x3 (HSM violation) (Errors) Jun 27 21:43:19 UNRAID kernel: ata12.00: status: { ERR } (Drive related) Jun 27 21:43:19 UNRAID kernel: ata12.00: error: { ABRT } (Errors) Jun 27 21:43:19 UNRAID kernel: ata12: hard resetting link (Minor Issues) Jun 27 21:43:19 UNRAID kernel: sas: sas_form_port: phy5 belongs to port1 already(1)! (Drive related) Jun 27 21:43:21 UNRAID kernel: drivers/scsi/mvsas/mv_sas.c 1701:mvs_I_T_nexus_reset for device[1]:rc= 0 (System) Jun 27 21:43:21 UNRAID kernel: sas: sas_ata_hard_reset: Found ATA device. (Drive related) Jun 27 21:43:21 UNRAID kernel: ata12.00: configured for UDMA/133 (Drive related) Jun 27 21:43:21 UNRAID kernel: ata12.00: device reported invalid CHS sector 0 (Drive related) Jun 27 21:43:21 UNRAID kernel: ata12: EH complete (Drive related) Jun 27 21:43:21 UNRAID kernel: ata13: sas eh calling libata port error handler (Routine) Jun 27 21:43:21 UNRAID kernel: ata14: sas eh calling libata port error handler (Routine) Jun 27 21:43:21 UNRAID kernel: sas: --- Exit sas_scsi_recover_host (Drive related) I searched for the errors and saw that some other people suggested that I un-bundle the sata cables, because I totally have mine zip-tied together right now. So I think I am going to open up my case and start to cut off the zip-ties. any other suggestions?
June 28, 201214 yr Only shutdown using the unRAID webGUI. Unclean shutdowns can definitely cause file system corruption like this. Attach the entire syslog. Zip if needed. Sata cables can be loosely bundled together. Do not bundle SATA with power.
June 29, 201214 yr Author Hello Dgaschk, Ok, here is what I have done so far: 1. Opened up the server, cut off all the zip ties to the sata cables. 2. Re-seated all the drives. 3. Re-seated all cables, one at a time to preserve cabling/port order. 4. I powered up the array, and started doing some heavy I/O, I was attempting to repair a large file with quickpar, I also loaded ember media manager which spun up all disks and started looking for media files. So far throughout all this, no errors, everything appears to be running stable. Then... I launch a preclear script for one of my new drives, sure enough in a few minutes, I start having problems, the web gui, becomes and stays unavailable, there is a hiccup in SMB shares, and my quickpar repair fails. However throughout this time Telnet access remains ok, and SMB access comes back so I was able to grab the syslog. I posted it up on paste bin here: http://pastebin.com/7GpQX7kR Thanks for all your help!
June 29, 201214 yr Need to determine if an add-on is causing the problem. Make the go file appear as in my sig. Just add # to the beginning of any other lines. Then restart and try to reproduce.
June 29, 201214 yr Author Hello Dgaschk, I did as you have asked, and commented out any running lines to match the go script in your signature. I then rebooted and unraid came up, and I double checked and UNmenu, etc were not running. I then attempted a preclear script on my drives again, and after 25 min, they froze, and the unraid webgui, was no longer available. I went ahead and decided to grab the syslog again, just in case you wanted to take a look: http://pastebin.com/2Tu1C43A As always I appreciate your help. Thank You
June 29, 201214 yr See here http://lime-technology.com/forum/index.php?topic=20529.0 Post a link to this thread in that tread.
June 29, 201214 yr Author Well, for the time being I have migrated back to Beta 14, as others have suggested that it does not have this issue. I am hoping that my problems with beta 14 were that I had a corrupt rieser FS on my cache drive, and now that, that is fixed, going back to it, I will hopefully experience solid performance again. Thank you
July 2, 201214 yr I may be experiencing a similar problem with RC5-R8168 (hard to tell remotely - left my house an hour ago with four preclear screens running at the 20-hour mark, everything seemed fine, but just checked and Telnet was killed / unable to reconnect...) Full details as soon as I'm home and can reboot the system, but stats preview for the next few hours: brand new box, see Raj's most recent 20-drive budget tower build, but only one SASLP-MV8 card right now (just in case I need to scrap the SAS and pick up some LSI cards instead...)
July 2, 201214 yr Author Just wanted to come back and say that reverting back to Beta 14 did indeed fix my problems, after I repaired my reiserFS issues. I was successfully able to pre-clear, and everything has been running great again. Thank you dgaschk! And hopefully this will be resolved in the final release. Thank you
Archived
This topic is now archived and is closed to further replies.