July 30, 200916 yr Nope. Everything on that drive can go. Thanks for asking~ Ok, you can completely re-partition that drive by typing the following three lines exactly as shown here sfdisk /dev/sdh <<EOF ; EOF Note, that is "sfdisk" (with a leading "s") The second line is a single ";" semicolon. Also, the prompt for the second and third lines will change to a ">" so don't be surprised, it is normal until you complete the three line sequence of commands. The third line is the three letters "EOF" It will take a few seconds... It will look a lot like this when I did it on drive /dev/hdi in my array: [pre]# sfdisk /dev/hdi <<EOF > ; > EOF Checking that no-one is using this disk right now ... OK Disk /dev/hdi: 48641 cylinders, 255 heads, 63 sectors/track Old situation: Units = cylinders of 8225280 bytes, blocks of 1024 bytes, counting from 0 Device Boot Start End #cyls #blocks Id System /dev/hdi1 0+ 48641- 48642- 390711352+ 0 Empty start: (c,h,s) expected (0,1,1) found (0,0,0) end: (c,h,s) expected (1023,254,63) found (0,0,0) /dev/hdi2 0 - 0 0 0 Empty /dev/hdi3 0 - 0 0 0 Empty /dev/hdi4 0 - 0 0 0 Empty New situation: Units = cylinders of 8225280 bytes, blocks of 1024 bytes, counting from 0 Device Boot Start End #cyls #blocks Id System /dev/hdi1 0+ 48640 48641- 390708832 83 Linux /dev/hdi2 0 - 0 0 0 Empty /dev/hdi3 0 - 0 0 0 Empty /dev/hdi4 0 - 0 0 0 Empty Warning: no primary partition is marked bootable (active) This does not matter for LILO, but the DOS MBR will not boot this disk. Successfully wrote the new partition table Re-reading the partition table ... If you created or changed a DOS partition, /dev/foo7, say, then use dd(1) to zero the first 512 bytes: dd if=/dev/zero of=/dev/foo7 bs=512 count=1 (See fdisk( 8 ) .) [/pre] Then, we can copy the entire disk5 to the first partition of the temp disk by typing: dd if=/dev/md5 of=/dev/sdh1 bs=2048k Note, we copy to /dev/sdh1 (with a "1" on the end) as it represents the 1st partition. Expect the copy to take an hour or two, or three, or more. Remember, it is reconstructing the data by reading all the other disks. it all depends on the speed of your array. The time it takes will probably be close to how long it takes to do a parity check. Now, once we do that we can always restore back to where we were, as we have a saved copy. Joe L.
July 30, 200916 yr Author root@Tower:/# sfdisk /dev/sdh <<EOF > ; > EOF Checking that no-one is using this disk right now ... OK Disk /dev/sdh: 121601 cylinders, 255 heads, 63 sectors/track Old situation: Units = cylinders of 8225280 bytes, blocks of 1024 bytes, counting from 0 Device Boot Start End #cyls #blocks Id System /dev/sdh1 * 0+ 12747 12748- 102398278+ 7 HPFS/NTFS /dev/sdh2 12748 121599 108852 874353690 f W95 Ext'd (LBA) /dev/sdh3 0 - 0 0 0 Empty /dev/sdh4 0 - 0 0 0 Empty /dev/sdh5 12748+ 121599 108852- 874353658+ 7 HPFS/NTFS New situation: Units = cylinders of 8225280 bytes, blocks of 1024 bytes, counting from 0 Device Boot Start End #cyls #blocks Id System /dev/sdh1 0+ 121600 121601- 976760032 83 Linux /dev/sdh2 0 - 0 0 0 Empty /dev/sdh3 0 - 0 0 0 Empty /dev/sdh4 0 - 0 0 0 Empty Warning: no primary partition is marked bootable (active) This does not matter for LILO, but the DOS MBR will not boot this disk. Successfully wrote the new partition table Re-reading the partition table ... If you created or changed a DOS partition, /dev/foo7, say, then use dd(1) to zero the first 512 bytes: dd if=/dev/zero of=/dev/foo7 bs=512 count=1 (See fdisk(.) Then, we can copy the entire disk5 to the first partition of the temp disk by typing: dd if=/dev/md5 of=/dev/sdh1 bs=2048k Did you mean if=/dev/md3? It was disk 3 that was the failed, although in the previous steps we had unassigned it. Thanks.
July 30, 200916 yr Sorry... yes... if it was disk3 that failed, then I do mean /dev/md3. (it was someone else with a bad disk5) SO dd if=/dev/md3 of=/dev/sdh1 bs=2048k
July 31, 200916 yr Author Looks like md3 only appears when the array is online. I moved online, and ran the copy command. It looks like it is getting the data from the parity since I noticed 'reads' on all disk. I have my case fans on to max, and corrected the earlier case fan failure so over heating shouldn't be a problem. I'll give you all an update when it's complete, or you can give me instructions for the next steps afterwards Thanks.
July 31, 200916 yr Looks like md3 only appears when the array is online. I moved online, and ran the copy command. It looks like it is getting the data from the parity since I noticed 'reads' on all disk. I have my case fans on to max, and corrected the earlier case fan failure so over heating shouldn't be a problem. I'll give you all an update when it's complete, or you can give me instructions for the next steps afterwards Thanks. Yes, that is the expected result... It is using parity and the other disks to reconstruct what it thinks might be on disk3. (It is just reconstructing bits... it has no knowledge of files... to the "dd" command, it is just each bit in turn.) Once the copy of the data to the added disk is complete, we can re-assign the physical disk on the devices page. It will still show up as un-formatted when you start the array back up. DO NOT FORMAT IT. Instead, run reiserfsck on /dev/md3 with the --rebuild-sb option. If there is anything there, it will try to fix it. The commands would be: cd samba stop umount /mnt/disk3 reiserfsck --rebuild-sb /dev/md3 I don't think the md3 drive is mounted, so it very likely the "umount" command will fail, don't worry if it complains. Let's see what the reiserfsck command does. The array will need to be running while you do all this. It might suggest a next action to take... let me know what it says before you follow its instructions. If it looks like the superblock is fixed, we can then reboot the server to re-mount the disk (easiest) or mount the disk back on /mnt/disk3, restart SAMBA, and see what it looks like on the LAN. At any time, going forward, we can get back to the point where you copied the contents of the failed disk to the temp disk by typing dd if=/dev/sdh1 of=/dev/md3 bs=2048k It will copy the old contents back and you would be able to try something else. (not sure what else, but we don't know what is on disk3 at all at this point. But, you have a copy of it saved.) Joe L.
July 31, 200916 yr Author Thanks Joe. The copy has completed and I've reassigned disk 3 (array had to be offline first) However, to go online again, I think I have to either "Start will bring the array on-line, start Data-Rebuild, and then expand the file system (if possible)." which I think I DO NOT want to do. Instead, I think I have to hit the restore command again. Currently it says: "Stopped. Disabled disk replaced." I wanted to double check with you first. Thanks. -Daniel
July 31, 200916 yr Let's think this through... it will try to rebuild the drive... but it won't be successful at the end... We've been there before. If we stay off-line we have two copies of the file-system where we can attempt a rebuild. If we press "restore" and set the invalid slot to be 99 with the mdcmd command, and then start the array it will just start on it check of the parity disk. It will still show "unformatted" as it will be unable to mount the disk. But... we can run the reiserfsck on it. This is probably the best choice. Therefore, I think you are correct. Leave disk3 assigned. 1. Press the button labeled "restore" which will invalidate parity. 2. enter the following commands to make it think parity is good (as good as we've got right now) cd mdcmd set invalidslot 99 3. Press the "Start" button to start the array. 4. Cancel the resulting parity check 5. Run the reiserfsck command to attempt to rebuild the superblock of the file system cd samba stop umount /mnt/disk3 reiserfsck --rebuild-sb /dev/md3 The file-system check will prompt you for some criteria. The answers must be supplied correctly, or it will not work. See the wiki http://lime-technology.com/wiki/index.php/Check_Disk_Filesystems It will point you to these threads: http://lime-technology.com/forum/index.php?topic=1483 and http://lime-technology.com/forum/index.php?topic=4021 They describe the correct responses. Be careful, at least one response is not the default response. What do you think? Again the umount will probably fail, as the disk is not mounted. Joe L.
August 1, 200916 yr Author root@Tower:~# reiserfsck --rebuild-sb /dev/md3 reiserfsck 3.6.19 (2003 www.namesys.com) ************************************************************* ** If you are using the latest reiserfsprogs and it fails ** ** please email bug reports to [email protected], ** ** providing as much information as possible -- your ** ** hardware, kernel, patches, settings, all reiserfsck ** ** messages (including version), the reiserfsck logfile, ** ** check the syslog file for any related information. ** ** If you would like advice on using this program, support ** ** is available for $25 at www.namesys.com/support.html. ** ************************************************************* Will check superblock and rebuild it if needed Will put log info to 'stdout' Do you want to run this program?[N/Yes] (note need to type Yes if you do):Yes reiserfs_open: the reiserfs superblock cannot be found on /dev/md3. what the version of ReiserFS do you use[1-4] (1) 3.6.x (2) >=3.5.9 (introduced in the middle of 1999) (if you use linux 2.2, choose this one) (3) < 3.5.9 converted to new format (don't choose if unsure) (4) < 3.5.9 (this is very old format, don't choose if unsure) (X) exit 1 Enter block size [4096]: 4096 No journal device was specified. (If journal is not available, re-run with --no-journal-available option specified). Is journal default? (y/n)[y]: y Did you use resizer(y/n)[n]: n rebuild-sb: no uuid found, a new uuid was generated (9ed31588-eec8-4f4c-b623-37dddd191b15) rebuild-sb: You either have a corrupted journal or have just changed the start of the partition with some partition table editor. If you are sure that the start of the partition is ok, rebuild the journal header. Do you want to rebuild the journal header? (y/n)[n]: y Reiserfs super block in block 16 on 0x903 of format 3.6 with standard journal Count of blocks on the device: 122096624 Number of bitmaps: 3727 Blocksize: 4096 Free blocks (count of blocks - used [journal, bitmaps, data, reserved] blocks): 0 Root block: 0 Filesystem is NOT clean Tree height: 0 Hash function used to sort names: not set Objectid map size 0, max 972 Journal parameters: Device [0x0] Magic [0x0] Size 8193 blocks (including 1 for journal header) (first block 18) Max transaction length 1024 blocks Max batch size 900 blocks Max commit age 30 Blocks reserved by journal: 0 Fs state field: 0x1: some corruptions exist. sb_version: 2 inode generation number: 0 UUID: 9ed31588-eec8-4f4c-b623-37dddd191b15 LABEL: Set flags in SB: Is this ok ? (y/n)[n]: y The fs may still be unconsistent. Run reiserfsck --check. root@Tower:~# Note: The program ran and finished with the "The fs may still be unconsistent. Run reiserfsck --check." within like 1-2 seconds. It was way too fast for comfort. (Well worst case, I can try to rebuild it from parity again. Last time we fixed the flash drive w/ Unraid after the rebuild, so that could have contributed to some of the issues).
August 1, 200916 yr Next step is to rebuild the bitmaps that point to the files. reiserfsck –scan-whole-partition –rebuild-tree /dev/md3
August 1, 200916 yr Author Thanks Joe~ Running now. I'll take 2 hours though. However, I might have some trouble pasting the results here. Since telnet to the server doesn't work any more. I had to manually enter the commands into the screen. I've taken a screenshot with a camera, but there are limited number of lines. I wonder if we can enable samba again, so we can look at the log after. hmmm.
August 2, 200916 yr Author Ok it finished running. Looks like it found a ton of files 100,000 files and 4000 directories. Sorry, I don't have the output as it was printed on the server screen, unless stdout log is filed some where.
August 2, 200916 yr Ok it finished running. Looks like it found a ton of files 100,000 files and 4000 directories. Sorry, I don't have the output as it was printed on the server screen, unless stdout log is filed some where. This is sounding encouraging. Now, you can try the following. mount -t reiserfs -o noacl,nouser_xattr,noatime,nodiratime /dev/md3 /mnt/disk3 cd /root samba start You should then be able to see the contents of the disk via windows explorer. There will probably be a lost+found folder. In it are all the files/folders it found where it could not determine the name. For most sub-directories, it should have been able to name the sub-directory properly. The process of scanning the "whole-partition" would have also caused it to grab files recently deleted, or at least those that had not yet been overwritten by other files... Don't be surprised to find stuff you had deleted. Joe L.
August 2, 200916 yr I was reading the output from when you rebuilt the superblock on the file-system. The last line it printed suggested you re-run reiserfsck --check /dev/md3 You might want to do that at some point... (You will need to stop samba, and un-mount the drive as described in the wiki) For now, let's see if it can be mounted, and what it was able to recover. Joe L.
August 2, 200916 yr Author root@Tower:~# mount -t reiserfs -o noacl,nouser_xattr,noatime,nodiratime /dev/md3 /mnt/disk3 mount: mount point /mnt/disk3 does not exist root@Tower:~# cd .. root@Tower:/# cd mnt root@Tower:/mnt# ls disk1/ disk2/ disk4/ disk5/ disk6/ disk7/
August 2, 200916 yr Author I ran the reiserfsck in the mean time. Looks pretty good. root@Tower:/# reiserfsck --check /dev/md3 reiserfsck 3.6.19 (2003 www.namesys.com) ************************************************************* ** If you are using the latest reiserfsprogs and it fails ** ** please email bug reports to [email protected], ** ** providing as much information as possible -- your ** ** hardware, kernel, patches, settings, all reiserfsck ** ** messages (including version), the reiserfsck logfile, ** ** check the syslog file for any related information. ** ** If you would like advice on using this program, support ** ** is available for $25 at www.namesys.com/support.html. ** ************************************************************* Will read-only check consistency of the filesystem on /dev/md3 Will put log info to 'stdout' Do you want to run this program?[N/Yes] (note need to type Yes if you do):Yes ########### reiserfsck --check started at Sat Aug 1 22:17:36 2009 ########### Replaying journal.. Reiserfs journal '/dev/md3' in blocks [18..8211]: 0 transactions replayed Checking internal tree..finished Comparing bitmaps..finished Checking Semantic tree: finished No corruptions found There are on the filesystem: Leaves 127181 Internal nodes 845 Directories 4424 Other files 115690 Data block pointers 121774809 (69 of them are zero) Safe links 0 ########### reiserfsck finished at Sat Aug 1 22:24:41 2009 ########### root@Tower:/#
August 2, 200916 yr I guess the mount-point directory was deleted when the mount failed. You can create it yourself by typing: mkdir /mnt/disk3 Then proceed with the mount as instructed in the prior post. Good that the file-system check seems clean now. Joe L.
August 2, 200916 yr Actually, at this point, you can probably stop the array and reboot. unRAID should be able to mount the repaired file-system on disk3 on its own. It will create the missing directory. (an empty directory is used as the mount point in Linux). Once you look around a bit, you might want to do a manual "parity check" by pressing the "Check" button on the main unRAID management web-page. I don't think the parity-check will find much, but I'll feel better knowing it was run, and all is back in sync after disk3 was properly mounted. Joe L.
August 2, 200916 yr Author Woot, looks like my files are there and working. Hmm, it wont' let me do a check, just a regular sync. I guess I'm pretty happy with the result after checking some of the more critical files and I can try different things with the saved data on the 1T hard drive. Thanks for all your help Joe, i'm indebted.
August 2, 200916 yr Woot, looks like my files are there and working. Hmm, it wont' let me do a check, just a regular sync. Not sure what you meant by that, but if it did a full parity check it is what I wanted. I guess I'm pretty happy with the result after checking some of the more critical files and I can try different things with the saved data on the 1T hard drive. I'm very happy if you got ANY of your files back.... Please, make a backup ... a copy on something else, an other PC, a writable DVD, or multiple DVDs of the critical files. Things you cannot replace if your server was hit by lightning/fire/flood/tornado, where reiserfsck won't help ... (family photos, records, important documents) This was painful enough. Thanks for all your help Joe, i'm indebted. You are welcome... I don't know if you'll be able to do any better with recovering more from the 1TB drive we made a copy onto. You can try running reiserfsck on /dev/sdh1 You'll have to create your own empty directory to mount it on after you run reiserfsck on it, and it will be outside of the protected array, but you can create a mount-point directory in /mnt. mkdir /mnt/temp mount -t reiserfs -o noacl,nouser_xattr,noatime,nodiratime /dev/sdh1 /mnt/temp Whatever you do, don't copy sdh1 back to /dev/md3 now, it would be back to having no superblock.... and don't press the "restore" button. ;) Joe L.
August 2, 200916 yr Author Total lost and found has about 3 files, 21.7MB of data. I've been through over 500 files, and all of them seem fine. The drive was completely full and the most important data makes up like maybe 1 gig's worth (they appear to be fine). I'll probably burn them onto DVD, and keep them off site (or use online back up) for those ones i guess.
Archived
This topic is now archived and is closed to further replies.