October 24, 201213 yr I'll try to make this short. I recently moved and was extremely careful of my server when doing so, as I always am. I left it unplugged for about 3 weeks. I plugged it in two days ago and stared to watch some shows. I checked the unraid interface and everything was good (all green). Last night I checked the interface and saw my parity drive red and disabled. I tried to update the parity but when it tried I just got tons of write errors and then it would give up. I decided to power down and check the cables. Powered up and one of my other drives went missing. Powered down got that drive back but lost another. Long story short, I now my parity drive is invalid and a data drive is now showing as unformatted. Question is, now what do I do? Is the data still on the unformatted drive? I've attached some of my system logs. One had a lot of repeating unimportant data so I trimmed it. Currently the server is powered down at home. I can get a current log when I get home and post it syslog-2012-10-23.txt syslog-2012-10-24.txt
October 24, 201213 yr Author Here's the current status of my server. Parity invalid and disk 7 unformatted. I bought new cables and installed them on disk 7 and parity as a precaution. Attached is my syslog from start up till now. Current status of unraid's main page and the smart report for my parity and disk 7. Update: Ran: reiserfsck --check /dev/md7 Output: root@Tower:~# reiserfsck --check /dev/md7 reiserfsck 3.6.21 (2009 www.namesys.com) ************************************************************* ** If you are using the latest reiserfsprogs and it fails ** ** please email bug reports to [email protected], ** ** providing as much information as possible -- your ** ** hardware, kernel, patches, settings, all reiserfsck ** ** messages (including version), the reiserfsck logfile, ** ** check the syslog file for any related information. ** ** If you would like advice on using this program, support ** ** is available for $25 at www.namesys.com/support.html. ** ************************************************************* Will read-only check consistency of the filesystem on /dev/md7 Will put log info to 'stdout' Do you want to run this program?[N/Yes] (note need to type Yes if you do):Yes The problem has occurred looks like a hardware problem. If you have bad blocks, we advise you to get a new hard drive, because once you get one bad block that the disk drive internals cannot hide from your sight,the chances of getting more are generally said to become much higher (precise statistics are unknown to us), and this disk drive is probably not expensive enough for you to you to risk your time and data on it. If you don't want to follow that follow that advice then if you have just a few bad blocks, try writing to the bad blocks and see if the drive remaps the bad blocks (that means it takes a block it has in reserve and allocates it for use for of that block number). If it cannot remap the block, use badblock option (-B) with reiserfs utils to handle this block correctly. bread: Cannot read the block (2): (Input/output error). Aborted root@Tower:~# Update 2: I also just bought a 3TB drive, assuming I can't trust my disk 7 any logger and feel a little unsure with my current parity. But the thing is: A) this is my first and only 3TB, so it will have to be parity. B) I'm on 4.7. Is it possible for me to assign the 3tb to parity and unraid only see and use 2tb? I do plan on upgrading to 5.0 relatively soon but not right yet and I have no other 3tb so I don't need my parity to be any bigger then 2tb anyhow. syslog-2012-10-24latest.txt disk7.txt parity.txt
October 25, 201213 yr Just learning myself, but I can relay something I read in another post. It was recommended to "resize the 3TB to 2TB using HPA, and rebuild the failed drive onto it", because they (like you) are running 4.7. Don't know if this will help or not, but hopefully its of use for you.
October 25, 201213 yr With 2 drives out of service the contents are not recoverable within unRAID. Pull disk7 and use a reiserfs recovery program on a PC.
October 26, 201213 yr Author Well I have been searching since I first got this problem and this is what I have uncovered: http://lime-technology.com/wiki/index.php/Check_Disk_Filesystems According to the above wiki I can run reiserfsck with these options: --fix-fixable --rebuild-sb --rebuild-tree I would like some advice on which option to run reiserfsck with. Also, should I make a clone of my disk7 to a spare disk (outside of unraid) encase one of those attempts with reiserfsck causes more harm then good? Can I remove my disk 7 and work on recovering it in another unraid server, so I can remove it from my primary server and start building parity to protect my remaining disks? According to Joe L. : I'm stuck with this reply from the utility. The problem has occurred looks like a hardware problem. If you have bad blocks, we advise you to get a new hard drive, because once you get one bad block that the disk drive internals cannot hide from your sight,the chances of getting more are generally said to become much higher (precise statistics are unknown to us), and this disk drive is probably not expensive enough for you to you to risk your time and data on it. If you don't want to follow that follow that advice then if you have just a few bad blocks, try writing to the bad blocks and see if the drive remaps the bad blocks (that means it takes a block it has in reserve and allocates it for use for of that block number). If it cannot remap the block, use badblock option (-B) with reiserfs utils to handle this block correctly. bread: Cannot read the block (2): (Input/output error). RMA Time? Possibly, but the inability to read block 2 could just as easily be because either the power or data cable is loose. Have you powered down and re-seated them? Joe L. Unfortunately I ran into other problems. First, I ran smartctl (had to install the package, as I'm on 4.2.1). That seemed to be OK, so I ran reiserfsck, but got a "Bad root block" error. (The two parts - my wrong typing included - are in the attachment.) At this point I was stumped - the disk seemed to be OK, but reiserfsck seemed to be stuck. So I went for Plan B, and replaced that disk 6 with my other 400GB Seagate and let unRaid recreate the data on the disk. I rebooted the server after that and the syslog (attached) seemed to show that it mounted OK, complete with a filesystem. So parity seemed to have rescued the day, but.... the Windows GUI showed the new disk 6 as unformatted. I rebooted again, same result. So I ran reiserfsck again (on the new disk, of course) and got the same "Bad root block" error. This seems to indicate that the problem with the disk (possibly the filesystem structure) already existed the last time the old drive was written to (and therefore recorded by the parity drive). Since the error message indicated that --rebuild-tree did not complete, I ran reiserfsck with this option and got some messages but nothing that seemed fatal. I rebooted again, and this time disk 6 was green-dotted. A quick check on the disk showed that the files seemed to be OK, and there was nothing in the lost+found directory, so everything seems to be OK. Ironically if I'd run "--rebuild-tree" on the old disk 6, I'd probably have fixed the disk - at least to the same extent. After this, I ran a parity check and then upgraded to 4.4.2. Now to add some more capacity..... Thanks for all the patient help on this, Rob and Joe. More suggestions for running reiserfsck: http://lime-technology.com/forum/index.php?topic=11648.0 http://lime-technology.com/forum/index.php?topic=21984.0
October 26, 201213 yr Basically, You run reiserfsck --check /dev/mdX (where md1 = disk1, md2 = disk2, md3 = disk3, etc) If reiserfsck complains about a missing superblock, seek help here before attempting to re-construct it. You can do more harm than good. You can certainly make a clone of a disk if you think it might help in a recovery effort. If you cannot access the /dev/mdX devices and are forced to run reiserfsck on the /dev/sdX1 devices, make sure you run it on the first partition, not the base drive. (/dev/sdx1, not /dev/sdx) Make sure you have the trailing "1" indicating you want to run it on the first partition. You ONLY run the --fix-fixable, --rebuild-sb, or --rebuild-tree if instructed by a prior --check. The output of a reiserfsck tells you how to proceed. In the rare case, even you need to rebuild a superblock, be aware the default settings to its prompts ARE NOT correct for unRAID. See the wiki for the correct settings. (I think they are described there) You'll need to stop processes and un-mount the drives before running reiserfsck in any "fix" mode. Those instructions are in the wiki link you gave in the prior post.
October 26, 201213 yr Author Ok, maybe I'm just losing it but my parity is orange and it says my parity is invalid in unmenu. I moved my disk 7 to a different controller to troubleshoot. Now my server is in this state (attached). What do I do?
October 26, 201213 yr Author I received bread: Cannot read the block (2): (Input/output error). when I ran: reiserfsck --check /dev/md7 I seem to be in the same situation as snsumner in this thread: http://lime-technology.com/forum/index.php?topic=11648.0 But according to the wiki: Important Note!!! Do NOT run reiserfsck with the --rebuild-sb or --rebuild-tree switches, unless you are instructed to, by the instruction of a previous run of reiserfsck, or by an expert user! So don't want to try anything until an expert tells me its a good idea. snsumner ran the options in this order: --rebuild-tree --rebuild-sb badblocks -nvs /dev/sdb Result: So overnight I ran badblock and long smarttest and both completed with no errors! To troubleshoot further I moved the disk to another controller. I then went into UnRAID and assigned the drive back to disk4. Now unraid is telling me its a new disk and giving me the option:
October 26, 201213 yr I received bread: Cannot read the block (2): (Input/output error). when I ran: reiserfsck --check /dev/md7 I seem to be in the same situation as snsumner in this thread: http://lime-technology.com/forum/index.php?topic=11648.0 But according to the wiki: Important Note!!! Do NOT run reiserfsck with the --rebuild-sb or --rebuild-tree switches, unless you are instructed to, by the instruction of a previous run of reiserfsck, or by an expert user! So don't want to try anything until an expert tells me its a good idea. snsumner ran the options in this order: --rebuild-tree --rebuild-sb badblocks -nvs /dev/sdb Result: So overnight I ran badblock and long smarttest and both completed with no errors! To troubleshoot further I moved the disk to another controller. I then went into UnRAID and assigned the drive back to disk4. Now unraid is telling me its a new disk and giving me the option: Basically, if block2 of the file-system cannot be read, the disk has stopped responding. There is no need to run any other reiserfsck command (and you were not instructed by a prior reiserfsck --check t run any) Either a cable is loose, (power or data) or the disk controller died, or you have a bad disk, or a bad cable to the disk, or a power supply too small to power all your disks. Since the disk was not responding, I do not see how you got badblocks to pass... Post the SMART report from the disk. Make sure it is the correct disk. (The /dev/sdX device names can change from one boot to the next) Verify by model/serial number. Joe L.
October 26, 201213 yr Author Well I pulled the drive and moved it to my other unraid server. I just ran reiserfsck again. This is the output. I guess between when I ran reiserfsck the first time and when I grabbed the SMART report from the drive I must have fixed a loose cable. So it looks as if I should go ahead with the --rebuild-sb option. I assume it shouldn't matter that this drive is in a different server? And I assume it's still advisable to make a clone of the drive first? Thank you for your help!! EDIT: oops. I guess I was supposed to run: reiserfsck --check /dev/sda1 Its checking now root@Tower2:/lib# fdisk -l Disk /dev/sda: 500.1 GB, 500107862016 bytes 1 heads, 63 sectors/track, 15504336 cylinders Units = cylinders of 63 * 512 = 32256 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x00000000 Device Boot Start End Blocks Id System /dev/sda1 2 15504336 488386552+ 83 Linux Partition 1 does not end on cylinder boundary. root@Tower2:/lib# reiserfsck --check /dev/sda reiserfsck 3.6.21 (2009 www.namesys.com) ************************************************************* ** If you are using the latest reiserfsprogs and it fails ** ** please email bug reports to [email protected], ** ** providing as much information as possible -- your ** ** hardware, kernel, patches, settings, all reiserfsck ** ** messages (including version), the reiserfsck logfile, ** ** check the syslog file for any related information. ** ** If you would like advice on using this program, support ** ** is available for $25 at www.namesys.com/support.html. ** ************************************************************* Will read-only check consistency of the filesystem on /dev/sda Will put log info to 'stdout' Do you want to run this program?[N/Yes] (note need to type Yes if you do):Yes reiserfs_open: the reiserfs superblock cannot be found on /dev/sda. Failed to open the filesystem. If the partition table has not been changed, and the partition is valid and it really contains a reiserfs partition, then the superblock is corrupted and you need to run this utility with --rebuild-sb. EDIT2: Yeah! Everything looks good from what I can tell. root@Tower2:/lib# reiserfsck --check /dev/sda1 reiserfsck 3.6.21 (2009 www.namesys.com) ************************************************************* ** If you are using the latest reiserfsprogs and it fails ** ** please email bug reports to [email protected], ** ** providing as much information as possible -- your ** ** hardware, kernel, patches, settings, all reiserfsck ** ** messages (including version), the reiserfsck logfile, ** ** check the syslog file for any related information. ** ** If you would like advice on using this program, support ** ** is available for $25 at www.namesys.com/support.html. ** ************************************************************* Will read-only check consistency of the filesystem on /dev/sda1 Will put log info to 'stdout' Do you want to run this program?[N/Yes] (note need to type Yes if you do):Yes ########### reiserfsck --check started at Fri Oct 26 04:39:30 2012 ########### Replaying journal: Done. Reiserfs journal '/dev/sda1' in blocks [18..8211]: 0 transactions replayed Checking internal tree.. finished Comparing bitmaps..finished Checking Semantic tree: finished No corruptions found There are on the filesystem: Leaves 51852 Internal nodes 331 Directories 5047 Other files 71951 Data block pointers 44200889 (0 of them are zero) Safe links 0 ########### reiserfsck finished at Fri Oct 26 04:43:38 2012 ###########
October 26, 201213 yr Author Fixed!!!! Added the drive back on my primary server in the same slot. Ran initconfig and its now building parity! I don't know what the hell went on. I guess somethings got loose during the move. Any tips to enure everything it in tip top working order? I assume just run a long SMART test on all suspect drives?? EDIT: Not quite yet. I can't catch a break. Looks my disk3 is throwing out a ton of read errors. syslog was too big to attach. It's full of: Oct 26 01:17:09 Tower kernel: handle_stripe read error: 112728728/1, count: 1 Oct 26 01:17:09 Tower kernel: md: disk3 read error Oct 26 01:17:09 Tower kernel: handle_stripe read error: 112728736/1, count: 1 Oct 26 01:17:09 Tower kernel: md: disk3 read error Oct 26 01:17:09 Tower kernel: handle_stripe read error: 112728744/1, count: 1 Is it possible for a "bank" of my controller to go bad?? I have 6 onboard sata ports. The 3 on the left contain the 3 drives that have been giving me trouble. disk3,disk5, and disk 7. Motherboard is: Gigabyte EP45-UD3LR EDIT2: Solved, checked my disk3 with reiserfsck everything was fine. So I ran initconfig and started up the array.
Archived
This topic is now archived and is closed to further replies.