September 17, 201213 yr So, background: Needed a 3TB drive for another vm (SageTV - not important) and took my cache drive out of service for this purpose. To fill in for the cache drive, I decided to take a 1.5TB drive out of my array and use it. Here was my config before: Parity: 3TB 1st: 1.5TB 2nd: 3TB 3rd: 1.5TB 4th: 1.5TB Cache 3TB So, backed up the contents of cache (using SABnzbd, Couchpotato, and Sickbeard) to a disk in the array, stopped the array, changed slot 4 to unassigned and assigned the 1.5TB disk from slot 4 as cache. Hit start array and got the missing disk 4. No problems so far. Copied the data from the array (backup cache files) to the cache drive and things were good there as well (all services up and running). Removed the 3TB disk (old cache) and rebooted and wanted to make sure my parity was healthy. So, did a quick google and hit here: http://lime-technology.com/wiki/index.php/FAQ_remove_drive. So, stopped the array, dropped to SSH and typed initconfig and refreshed the web page. Now, all the slots were blank and I thought I remembered were the disks went, but I screwed up and didn't have a screen shot (later remembered I did have daily emails with md status that I used later) so I guessed and guessed wrong. When I hit start, it came up and said disk 2 was unformatted as well as told me it was updating parity (or something like that). I panic'd and hit stop array, and had to go back to ssh and kill all of the cache_dirs based find commands that were still running as the array had just started. It probably took me 30 seconds to get the array to stop, so I believe it was running for at most 2 minutes. So, instead of stopping here and asking for help, I moved parity and disk 2 around to their correct spots (now remembering that I had a status email with the drive serial numbers in it and slots). Again, pressed start array and disk 2 (now the correct drive) came up unformatted and the same message for updating parity. I again hit stop, killed cache_dirs, and hit the forums. I found this awesome forum post (which I had read a few weeks ago) http://lime-technology.com/forum/index.php?topic=1483.0 So, following along with that page as well as http://lime-technology.com/wiki/index.php/Check_Disk_Filesystems I started the array in maintenance mode, and started the command reiserfsck /dev/md2 since it's my disk2. root@Audi:/# reiserfsck /dev/md2 reiserfsck 3.6.21 (2009 www.namesys.com) ************************************************************* ** If you are using the latest reiserfsprogs and it fails ** ** please email bug reports to [email protected], ** ** providing as much information as possible -- your ** ** hardware, kernel, patches, settings, all reiserfsck ** ** messages (including version), the reiserfsck logfile, ** ** check the syslog file for any related information. ** ** If you would like advice on using this program, support ** ** is available for $25 at www.namesys.com/support.html. ** ************************************************************* Will read-only check consistency of the filesystem on /dev/md2 Will put log info to 'stdout' Do you want to run this program?[N/Yes] (note need to type Yes if you do):Yes reiserfs_open: the reiserfs superblock cannot be found on /dev/md2. Failed to open the filesystem. If the partition table has not been changed, and the partition is valid and it really contains a reiserfs partition, then the superblock is corrupted and you need to run this utility with --rebuild-sb. As expected, need to rebuild-sb root@Audi:/# reiserfsck --rebuild-sb /dev/md2 reiserfsck 3.6.21 (2009 www.namesys.com) ************************************************************* ** If you are using the latest reiserfsprogs and it fails ** ** please email bug reports to [email protected], ** ** providing as much information as possible -- your ** ** hardware, kernel, patches, settings, all reiserfsck ** ** messages (including version), the reiserfsck logfile, ** ** check the syslog file for any related information. ** ** If you would like advice on using this program, support ** ** is available for $25 at www.namesys.com/support.html. ** ************************************************************* Will check superblock and rebuild it if needed Will put log info to 'stdout' Do you want to run this program?[N/Yes] (note need to type Yes if you do):Yes reiserfs_open: the reiserfs superblock cannot be found on /dev/md2. what the version of ReiserFS do you use[1-4] (1) 3.6.x (2) >=3.5.9 (introduced in the middle of 1999) (if you use linux 2.2, ch oose this one) (3) < 3.5.9 converted to new format (don't choose if unsure) (4) < 3.5.9 (this is very old format, don't choose if unsure) (X) exit 1 Enter block size [4096]: 4096 No journal device was specified. (If journal is not available, re-run with --no- journal-available option specified). Is journal default? (y/n)[y]: Y No journal device was specified. (If journal is not available, re-run with --no- journal-available option specified). Is journal default? (y/n)[y]: y Did you use resizer(y/n)[n]: n rebuild-sb: no uuid found, a new uuid was generated (e0832665-d2e5-45e8-aef1-70b 8e34484a5) rebuild-sb: You either have a corrupted journal or have just changed the start of the partition with some partition table editor. If you are sure that the start of the partition is ok, rebuild the journal header. Do you want to rebuild the journal header? (y/n)[n]: y Reiserfs super block in block 16 on 0x902 of format 3.6 with standard journal Count of blocks on the device: 732566624 Number of bitmaps: 22357 Blocksize: 4096 Free blocks (count of blocks - used [journal, bitmaps, data, reserved] blocks): 0 Root block: 0 Filesystem is NOT clean Tree height: 0 Hash function used to sort names: not set Objectid map size 0, max 972 Journal parameters: Device [0x0] Magic [0x0] Size 8193 blocks (including 1 for journal header) (first block 18) Max transaction length 1024 blocks Max batch size 900 blocks Max commit age 30 Blocks reserved by journal: 0 Fs state field: 0x1: some corruptions exist. sb_version: 2 inode generation number: 0 UUID: e0832665-d2e5-45e8-aef1-70b8e34484a5 LABEL: Set flags in SB: Mount count: 1 Maximum mount count: 30 Last fsck run: Sun Sep 16 18:04:44 2012 Check interval in days: 180 Is this ok ? (y/n)[n]: y The fs may still be unconsistent. Run reiserfsck --check. followed by root@Audi:/# reiserfsck /dev/md2 reiserfsck 3.6.21 (2009 www.namesys.com) ************************************************************* ** If you are using the latest reiserfsprogs and it fails ** ** please email bug reports to [email protected], ** ** providing as much information as possible -- your ** ** hardware, kernel, patches, settings, all reiserfsck ** ** messages (including version), the reiserfsck logfile, ** ** check the syslog file for any related information. ** ** If you would like advice on using this program, support ** ** is available for $25 at www.namesys.com/support.html. ** ************************************************************* Will read-only check consistency of the filesystem on /dev/md2 Will put log info to 'stdout' Do you want to run this program?[N/Yes] (note need to type Yes if you do):Yes ########### reiserfsck --check started at Sun Sep 16 19:52:24 2012 ########### Replaying journal: Done. Reiserfs journal '/dev/md2' in blocks [18..8211]: 0 transactions replayed Zero bit found in on-disk bitmap after the last valid bit. Checking internal tree.. Bad root block 0. (--rebuild-tree did not complete) Aborted root@Audi:/# So, I've now started with reiserfsck --scan-whole-partition --rebuild-tree /dev/md2 root@Audi:/# reiserfsck --scan-whole-partition --rebuild-tree /dev/md2 reiserfsck 3.6.21 (2009 www.namesys.com) ************************************************************* ** Do not run the program with --rebuild-tree unless ** ** something is broken and MAKE A BACKUP before using it. ** ** If you have bad sectors on a drive it is usually a bad ** ** idea to continue using it. Then you probably should get ** ** a working hard drive, copy the file system from the bad ** ** drive to the good one -- dd_rescue is a good tool for ** ** that -- and only then run this program. ** ** If you are using the latest reiserfsprogs and it fails ** ** please email bug reports to [email protected], ** ** providing as much information as possible -- your ** ** hardware, kernel, patches, settings, all reiserfsck ** ** messages (including version), the reiserfsck logfile, ** ** check the syslog file for any related information. ** ** If you would like advice on using this program, support ** ** is available for $25 at www.namesys.com/support.html. ** ************************************************************* Will rebuild the filesystem (/dev/md2) tree Will put log info to 'stdout' Do you want to run this program?[N/Yes] (note need to type Yes if you do):Yes Replaying journal: Done. Reiserfs journal '/dev/md2' in blocks [18..8211]: 0 transactions replayed Zero bit found in on-disk bitmap after the last valid bit. Fixed. ########### reiserfsck --rebuild-tree started at Sun Sep 16 19:59:19 2012 ########### Pass 0: ####### Pass 0 ####### The whole partition (732566624 blocks) is to be scanned Skipping 30567 blocks (super block, journal, bitmaps) 732536057 blocks will be read 0%. left 523983340, 48332 /sec Now, I have the fear that I've compounded the problem by not asking for help before getting too far. Am I on the right path so far? I have a backup of the critical data (it was all on disk1 anyway so I'm sure family photos / videos and purchased music is safe. What I have is tons of recorded TV shows (many seasons of shows that I haven't caught up on yet) and ripped DVDs and .isos from old technet subscription. So it's not exactly irreplaceable but would like to recover what I can. I'm running 5.0rc5, with Simple Features installed if it makes a difference. The stats on the main page are: 5045 writes to parity (not sure if this is from reboot about 4 hours ago cleaning up some VM things or the last array start). So not sure how much of the real disk 2 I wiped out. Any advice on were to go from here. I plan on letting the reiserfsck with the rebuild options mentioned above continue since I'm not sure it's safe to abort anyways. Thanks for the awesome community and support. syslog-2012-09-16.zip
Archived
This topic is now archived and is closed to further replies.