July 26, 200916 yr Author Ok chkdisk complete. (I think it fixed some issues before too. When I took a look for config.old I noticed I had trouble copying it. Now I can) I plugged it back in, ls showed nothing, but 'mount boot' fixed it, and I can see the files again. Ready for file system check on disk 3.
July 26, 200916 yr The command sequence to check the file-system would normally be to un-mount the disk from the mount point (but your disk3 is not currently mounted, so don't need to do that) I can see from the vol_id commands you did earlier that the new physical disk you installed seems to have an "ntfs" file system on it. we might see the same when we look at it through the "md" device... Let's check first. Type vol_id /dev/md3 If it shows an "ntfs" file system, then a reiserfs file-system check might not have anything to work on. But, for not you can just try reiserfsck /dev/md3 Do not try any of the "fix" options to reiserfsck yet..., even if it suggests it as a next step. It assumes you are operating on an reiserfs file-system. It may not be... you might make things worse. We want to proceed in a read-only, non-destructive mode until we have no other choices, or have a firm idea an action will help. Joe L.
July 26, 200916 yr Author root@Tower:/# reiserfsck /dev/md3 reiserfsck 3.6.19 (2003 www.namesys.com) ************************************************************* ** If you are using the latest reiserfsprogs and it fails ** ** please email bug reports to [email protected], ** ** providing as much information as possible -- your ** ** hardware, kernel, patches, settings, all reiserfsck ** ** messages (including version), the reiserfsck logfile, ** ** check the syslog file for any related information. ** ** If you would like advice on using this program, support ** ** is available for $25 at www.namesys.com/support.html. ** ************************************************************* Will read-only check consistency of the filesystem on /dev/md3 Will put log info to 'stdout' Do you want to run this program?[N/Yes] (note need to type Yes if you do):Yes reiserfs_open: the reiserfs superblock cannot be found on /dev/md3. Failed to open the filesystem. If the partition table has not been changed, and the partition is valid and it really contains a reiserfs partition, then the superblock is corrupted and you need to run this utility with --rebuild-sb. root@Tower:/# (I get the feeling i'll need to format the disk in reiserfsck (w/o affecting parity), and do the invalid disk and rebuild again)
July 26, 200916 yr You can try typing debugreiserfs /dev/md3 The output will look a lot like this if a reiserfs exists: debugreiserfs /dev/md3 debugreiserfs 3.6.19 (2003 www.namesys.com) Filesystem state: consistency is not checked after last mounting Reiserfs super block in block 16 on 0x903 of format 3.6 with standard journal Count of blocks on the device: 122096638 Number of bitmaps: 3727 Blocksize: 4096 Free blocks (count of blocks - used [journal, bitmaps, data, reserved] blocks): 5558748 Root block: 84439059 Filesystem is NOT clean Tree height: 5 Hash function used to sort names: "r5" Objectid map size 36, max 972 Journal parameters: Device [0x0] Magic [0x45ef01ab] Size 8193 blocks (including 1 for journal header) (first block 18) Max transaction length 1024 blocks Max batch size 900 blocks Max commit age 30 Blocks reserved by journal: 0 Fs state field: 0x0: sb_version: 2 inode generation number: 19363 UUID: 338cef44-fbae-41f0-ae4b-76061086d16a LABEL: Set flags in SB: ATTRIBUTES CLEAN
July 26, 200916 yr Author Thanks Joe. root@Tower:/# debugreiserfs /dev/md3 debugreiserfs 3.6.19 (2003 www.namesys.com) reiserfs_open: the reiserfs superblock cannot be found on /dev/md3. debugreiserfs: can not open reiserfs on "/dev/md3": no filesystem found root@Tower:/#
July 26, 200916 yr (I get the feeling i'll need to format the disk in reiserfsck (w/o affecting parity), and do the invalid disk and rebuild again) NO.... there is no way or need to format the disk... If you do, you WILL lose the data you had on it. Your best bet right now is to stop the array, un-assign disk3, and attempt to re-start it. Un-assigning a disk is exactly the same as if it had failed. When un-assigned, the contents are simulated using parity and the other data disks. when you re-start the array, the un-assigned disk should show as "red" and we can see what, if anything the parity drive has regarding the old contents. I'm getting a quezy feeling that your earlier actions, before you made your first post asking for guidance may have already erased the prior knowledge of what was on disk3. Pressing the button labeled "restore" was a very bad thing to do when a failed disk is present. (equivalent to asking the array to forget anything it ever knew about the failed disk) I know you said you were unable to start the array, but all it would need is a few seconds to write to parity when the disk3 was un-mounted, erasing the reiser superblock memory of it in parity. Later, when re-constucting from parity, all you would get back is zeros... If it was erased, we can attempt to re-create it, but one step at a time... as I said, you have already done some damage, now need to determine how much. Joe L.
July 26, 200916 yr Thanks Joe. root@Tower:/# debugreiserfs /dev/md3 debugreiserfs 3.6.19 (2003 www.namesys.com) reiserfs_open: the reiserfs superblock cannot be found on /dev/md3. debugreiserfs: can not open reiserfs on "/dev/md3": no filesystem found root@Tower:/# That is NOT a good sign. Try un-assigning the disk as I said, and see if you can start the array. Again, do not let it calculate parity, do not press format... do not pass "go", do not collect $300. We can always re-assign the drive and then try the rebuild-superblock option on the drive... It is not any guarantee, but it all depends on how much of the file-system was clobbered after you pressed "restore" Joe L.
July 26, 200916 yr That is NOT a good sign. Try un-assigning the disk as I said, and see if you can start the array. Again, do not let it calculate parity, do not press format... do not pass "go", do not collect $300. We can always re-assign the drive and then try the rebuild-superblock option on the drive... It is not any guarantee, but it all depends on how much of the file-system was clobbered after you pressed "restore" Joe L. I am hopeful but not optimistic on this one. aomega - have you ever run a successful parity check (not the build, but a check)?
July 26, 200916 yr Author aomega - have you ever run a successful parity check (not the build, but a check)? Yes before the failed hard drive issue, I used to run it very often, and then once a month (it never detected an error ever), since then I may not have run it for a while (few months). @Joe I set it to unassigned, and I was able to restart it. It currently says Unformatted disk(s) present. Screen shot attached.
July 26, 200916 yr That's what I thought would happen actually, but not what I wanted to happen. When you rebuilt from parity and the other disks, by forcing the invalid slot to be disk 3, it built what it thought it had there. Unfortunately, it looks like you had in some previous attempt overwritten the parity information. It was probably when you pressed restore and re-started the array. At that point it would have been busy writing parity with the smaller number of disks present, especially if your cabling was not seated properly. I wonder if you actually have an ntfs file system. You could try (as an exercise) mkdir /tmp/disk3 mount -r -t ntfs /dev/md3 /tmp/disk3 and see if the read-only mount succeeds. If it does, I'd say your chances of data recover are getting very very small. If the mount does not complain, you can do an ls -l /tmp/disk3 and then umount /tmp/disk3 Joe L.
July 26, 200916 yr Author Thanks Joe. I'm not sure if this is good, but the mount failed. root@Tower:/# mount -r -t ntfs /dev/md3 /tmp/disk3 mount: unknown filesystem type 'ntfs' root@Tower:/#
July 26, 200916 yr Thanks Joe. I'm not sure if this is good, but the mount failed. root@Tower:/# mount -r -t ntfs /dev/md3 /tmp/disk3 mount: unknown filesystem type 'ntfs' root@Tower:/# It indicates your version of the unRAID software does not have the ntfs driver loaded by default. You do have a read-only driver though you can load. You might need to load the ntfs driver first by typing: modprobe ntfs followed by the mount command mount -r -t ntfs /dev/md3 /tmp/disk3
July 26, 200916 yr Author root@Tower:/# modprobe ntfs root@Tower:/# mount -r -t ntfs /dev/md3 /tmp/disk3 mount: wrong fs type, bad option, bad superblock on /dev/md3, missing codepage or other error In some cases useful info is found in syslog - try dmesg | tail or so root@Tower:/# Thanks~
July 26, 200916 yr root@Tower:/# modprobe ntfs root@Tower:/# mount -r -t ntfs /dev/md3 /tmp/disk3 mount: wrong fs type, bad option, bad superblock on /dev/md3, missing codepage or other error In some cases useful info is found in syslog - try dmesg | tail or so root@Tower:/# Thanks~ So it is not really an ntfs file system either.
July 26, 200916 yr Author If it helps with debugging, the replacement hard drive was newly purchased. I formatted in Windows NTFS but did not create partitions.
July 27, 200916 yr Yes before the failed hard drive issue, I used to run it very often, and then once a month (it never detected an error ever), since then I may not have run it for a while (few months). That's good to hear. If anyone can get you out of this mess its Joe L. Good luck!
July 27, 200916 yr Do you still have the original 1TB drive you tried to use as a replacement for disk3? If so, do you have a spare disk controller port on your server? I'm trying to learn if you can add it, even if hanging off the side, so we can make an identical image of the failed drive current contents for safe-keeping while we see if it is possible to replace the superblock that is missing on the failed drive. If you do have a spare SATA controller port we can probably do this, if not, you will not be able to regroup to this same point................ Joe L.
July 27, 200916 yr Author Hey Joe, I do have one more free SATA port free (and another PCI controller card if necessary) and access to the 1T hard drive. While I can connect the 1T drive, I should probably turn off the computer first. Although I think the raid array will complain when it boots up. -Daniel
July 27, 200916 yr Hey Joe, I do have one more free SATA port free (and another PCI controller card if necessary) and access to the 1T hard drive. While I can connect the 1T drive, I should probably turn off the computer first. Although I think the raid array will complain when it boots up. -Daniel Don't do anything just yet, just exploring options. Yes, you should power down to add the new drive if you install it. The unRAID array should not care at all about the extra drive. The only way it would is if linux re-assigns drive IDs. In which case, you would just need to assign back your existing drives to their current assigned slots. (Take a screen-print of your devices page now)
July 27, 200916 yr Author Alright, i have a hard drive ready to add to the server, but I haven't done anything yet.
July 27, 200916 yr Ok, here is what I'm thinking. We have no idea what is on the drive you replaced the failing disk with. At best, it is your files, but at a minimum we know the superblock has been overwritten with zeros. (it is not recognized) At worst case, there is nothing that can be recovered at all. Before we try to rebuild the superblock, and since you have a spare 1TB disk we can use as temporary storage, we can copy the raw contents of the failed disk as reconstructed by parity to the spare disk outside of the array. This is probably exactly the same contents as on the reconstructed disk you used to replace the failed disk. Step 1. Stop the array Step 2. Power down Step 3. Install the spare 1TB disk on the spare port on the disk controller. step 4. Power up. Take note of the /dev/sd? that shows in the new syslog "device inventory" for the new disk. The array should still show disk3 missing. (we had un-assigned it yesterday) It is OK to start the array. (if it will start) Then, we will use a unix command to perform a raw data copy of the failed disk to the new disk you installed as temp space. I'll give you that command once we get this far. The reason for this backup copy is so we can operate on it to try to recover files rather than the original. (We can't do any more damage to the original this way... may never recover anything, but we don't burn bridges behind us) Take note of the new 1TB serial number now so you can recognize it later. Take a screen shot of the devices page now, so in case the MB re-assigns /dev/sd? letters when you add the new drive, you can re-assign the drives to their correct assignments on the devices page. Joe L.
July 28, 200916 yr Just a couple of observations (no solutions!), that may or may not help, but may provide some insight for Joe (or may just further confuse the issue!). Back in this post is this extract: root@Tower:/dev# vol_id /dev/sda1 ID_FS_TYPE=ntfs ID_FS_UUID=56309EA4309E8B1F root@Tower:/dev# vol_id /dev/sdb1 ID_FS_TYPE=ntfs ID_FS_UUID=56309EA4309E8B1F Device sdb appears to be linked to sda, same file system and UUID! Possible corruption? I would not do anything important without a reboot. Also, the syslog shows the rebuild of Disk 3, but NOT the invalidslot command and result. I suspect from the comments that when run earlier, it did show the expected results, but then the rebuild failed, so the system was rebooted(?), and then the rebuild proceeded, but not after a fresh invalidslot command had been executed. I have no idea whether this is important or not, whether the invalidslot command sets special internal conditions for the rebuild, that might be lost after reboot. This thread has gotten complicated enough, it would be nice to hear from Tom.
July 28, 200916 yr Author Hi Joe, I've installed the 1T drive. It used to be part of a windows computer with two partitions. root@Tower:/dev# fdisk -l sdh Disk sdh: 1000.2 GB, 1000204886016 bytes 255 heads, 63 sectors/track, 121601 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Device Boot Start End Blocks Id System sdh1 * 1 12748 102398278+ 7 HPFS/NTFS sdh2 12749 121600 874353690 f W95 Ext'd (LBA) sdh5 12749 121600 874353658+ 7 HPFS/NTFS root@Tower:/dev#
July 28, 200916 yr Hi Joe, I've installed the 1T drive. It used to be part of a windows computer with two partitions. root@Tower:/dev# fdisk -l sdh Disk sdh: 1000.2 GB, 1000204886016 bytes 255 heads, 63 sectors/track, 121601 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Device Boot Start End Blocks Id System sdh1 * 1 12748 102398278+ 7 HPFS/NTFS sdh2 12749 121600 874353690 f W95 Ext'd (LBA) sdh5 12749 121600 874353658+ 7 HPFS/NTFS root@Tower:/dev# Is there anything on that drive that you need? (can we overwrite its existing partitioning/data) ? Joe L.
Archived
This topic is now archived and is closed to further replies.