April 29, 200917 yr Joe L, This is my first time I have ever posted on a forum so hopefully the screenshot is attached . If not, it looks like it read all disks and wrote to disk 4. In summary Disk Size Free Reads Writes Errors parity 1,465,138,552 - 3,827,004 16 0 disk1 1,465,137,496 214,392,004 3,824,515 7 0 disk2 1,465,138,552 192,363,112 3,824,219 6 0 disk3 976,761,496 28,920,976 2,563,065 6 0 disk4 1,465,138,552 Unformatted 44 4,150,754 0 root@Tower:~# mount fusectl on /sys/fs/fuse/connections type fusectl (rw) usbfs on /proc/bus/usb type usbfs (rw) /dev/sda1 on /boot type vfat (rw,noatime,nodiratime,umask=0,shortname=mixed) /dev/md1 on /mnt/disk1 type reiserfs (rw,noatime,nodiratime,noacl,nouser_xattr) /dev/md2 on /mnt/disk2 type reiserfs (rw,noatime,nodiratime,noacl,nouser_xattr) /dev/md3 on /mnt/disk3 type reiserfs (rw,noatime,nodiratime,noacl,nouser_xattr) shfs on /mnt/user type fuse.shfs (rw,nosuid,nodev,allow_other) nfsd on /proc/fs/nfs type nfsd (rw) root@Tower:~# Excellent... (your screen attachment worked perfectly) The reason the disk is showing up as unformatted is because it is not mounted. That might be because you had used the empty disk.cfg, but I'm really not sure. Things are still looking in your favor so far. To know what unRAID did, before you do anything more, can we request a current copy of your syslog? Do not press "Restore" at this point. The reason you do not have a reboot or shutdown button is because the array is running. It must be stopped for those to appear. Let's see what the syslog shows. Also, can you do an fdisk -l /dev/sd? On the new disk4. (with the ? replaced with the correct drive letter) This will show if the disk is now partitioned Also, a vol_id /dev/sd?1 (again, replace the ? with the correct drive letter, and note there s a "1" at the end of sd?1, since you are asking it to report on the first partition. This should show if unRAID thinks there is a reiserfs file-system on the partition. None of these will write anything to the array, so they are all non-destructive things to try. Joe L.
April 29, 200917 yr Oh yes, DO NOT PRESS the "Format" button either... (It would un-do all you have done) Joe L.
April 29, 200917 yr Should a reboot not bring the drive up as formatted if there is a valid file system? Could this just related to the "unformatted" bug when unRAID is stopped while the drive is busy? When a drive replacement is done with unRAID under a normal exchange or failure, would unRAID not format the drive first before rebuilding the data on it? Does this not mean a format should have been done before using the mdcmd command and starting the array? Joe, you posted recently that the drive partition info was in the MBR and that parity is not calculated for that part of the drive. So, that means all the data is present but the rebuild never created the partition info so the drive is not recognized as formatted. So, will the drive not require formatting? Will a quick formatting setup the partition info but not mess with the data? You have worked with formatting and pre-clearing more than anybody. There must be a way to create a partition without changing any data in the partition area? Peter
April 29, 200917 yr Prior to the failure, had you ever run a full parity check to confirm that your parity was correct? If the parity info is not right, the rebuild will be garbage. But the parity can be totally messed up and the array work perfectly for months or years. Again, it is only if a disk fails that parity has to be correct. The only thing left to try is to run reiserfsck on the recovered disk. This should not be required in your situation as you explained it, so I am not particularly optimistic.
April 29, 200917 yr Should a reboot not bring the drive up as formatted if there is a valid file system? Could this just related to the "unformatted" bug when unRAID is stopped while the drive is busy?It might, but that bug occurs when one or more drive cannot be un-mounted. Those that can be un-mounted then show up as un-formatted. It is slightly different. When a drive replacement is done with unRAID under a normal exchange or failure, would unRAID not format the drive first before rebuilding the data on it? No, it does not format the drive. It reconstructs the MBR as needed based on the new drive size, then it uses parity and the other dirve data to supply the remaining contents. No formatting at all occurs. Does this not mean a format should have been done before using the mdcmd command and starting the array? No, as I said, it is not needed at all. Joe, you posted recently that the drive partition info was in the MBR and that parity is not calculated for that part of the drive. So, that means all the data is present but the rebuild never created the partition info so the drive is not recognized as formatted. Normally, emhttp takes care of that. I don't see why it would not in this case. It is an interesting thought... The output of the fdisk -ll and vol_id will tell us a lot more. So, will the drive not require formatting? You are confusing partitioning, with formatting. It might need the MBR set to define the first partition, but not formatting. Will a quick formatting setup the partition info but not mess with the data?It would erase the data. You have worked with formatting and pre-clearing more than anybody. There must be a way to create a partition without changing any data in the partition area? Yes. I really need to see the fdisk -l output, and the vol_id output before giving any advice on how to proceed. It might be as simple as a reboot, or as complicated as fixing the MBR. Joe L.
April 29, 200917 yr You are confusing partitioning, with formatting. It might need the MBR set to define the first partition, but not formatting. Yes, I am. The partition info isn't part of the data written during a rebuild so I'm thinking a partition still needs to be created. The unRAID code used during a normal drive failure and replacement must do more than just issue the mdcmd command because, as I understand the mdcmd command, it just tells unRAID to re-construct the data, not to handle replacing the drive. Using the default "mdcmd set invalidslot 0" is for parity and parity gets constructed without a partition. I'm thinking we just did the same thing as constructing parity but on a virgin data drive, ending up with a drive that's more or less the equivalent of a parity drive as seen by any OS. Of course, it could be as bjp999 posted and the parity data was bad.... Peter
April 30, 200917 yr Author (1) Here is the output from those two commands, it looks like the vol_id is missing from sdd1 so I think it's a safe assumption that it's was where the new drive is mounted. (2) The syslog is pretty big so I made it an attachment, hope that's ok! (3) I would say that I did a full parity rebuild a couple of weeks ago (~2). The server was rebooted since then and the disk was used many times since so I'm pretty sure it was ok when I did that. Tim ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ root@Tower:/dev# fdisk -l /dev/sdb Disk /dev/sdb: 1500.3 GB, 1500300828160 bytes 1 heads, 63 sectors/track, 46512302 cylinders Units = cylinders of 63 * 512 = 32256 bytes Disk identifier: 0x00000000 Device Boot Start End Blocks Id System /dev/sdb1 2 46512303 1465137496 83 Linux Partition 1 does not end on cylinder boundary. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ root@Tower:/dev# fdisk -l /dev/sdc Disk /dev/sdc: 1000.2 GB, 1000203804160 bytes 1 heads, 63 sectors/track, 31008302 cylinders Units = cylinders of 63 * 512 = 32256 bytes Disk identifier: 0x00000000 Device Boot Start End Blocks Id System /dev/sdc1 2 31008303 976761496 83 Linux Partition 1 does not end on cylinder boundary. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ root@Tower:/dev# fdisk -l /dev/sdd Disk /dev/sdd: 1500.3 GB, 1500301910016 bytes 1 heads, 63 sectors/track, 46512336 cylinders Units = cylinders of 63 * 512 = 32256 bytes Disk identifier: 0x00000000 Device Boot Start End Blocks Id System /dev/sdd1 2 46512336 1465138552+ 83 Linux Partition 1 does not end on cylinder boundary. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ root@Tower:/dev# fdisk -l /dev/sde Disk /dev/sde: 1500.3 GB, 1500301910016 bytes 1 heads, 63 sectors/track, 46512336 cylinders Units = cylinders of 63 * 512 = 32256 bytes Disk identifier: 0x00000000 Device Boot Start End Blocks Id System /dev/sde1 2 46512336 1465138552+ 83 Linux Partition 1 does not end on cylinder boundary. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ root@Tower:/dev# vol_id /dev/sdb1 ID_FS_USAGE=filesystem ID_FS_TYPE=reiserfs ID_FS_VERSION=3.6 ID_FS_UUID=40f1d9da-787f-4ef0-b40d-3da12c799803 ID_FS_UUID_ENC=40f1d9da-787f-4ef0-b40d-3da12c799803 ID_FS_LABEL= ID_FS_LABEL_ENC= ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ root@Tower:/dev# vol_id /dev/sdc1 ID_FS_USAGE=filesystem ID_FS_TYPE=reiserfs ID_FS_VERSION=3.6 ID_FS_UUID=429bfbfc-c121-4ca5-be2a-7ece60fb52ed ID_FS_UUID_ENC=429bfbfc-c121-4ca5-be2a-7ece60fb52ed ID_FS_LABEL= ID_FS_LABEL_ENC= ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ root@Tower:/dev# vol_id /dev/sdd1 /dev/sdd1: unknown volume type ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ root@Tower:/dev# vol_id /dev/sde1 ID_FS_USAGE=filesystem ID_FS_TYPE=reiserfs ID_FS_VERSION=3.6 ID_FS_UUID=034f732b-3aa6-476c-a26b-78a3fd599b72 ID_FS_UUID_ENC=034f732b-3aa6-476c-a26b-78a3fd599b72 ID_FS_LABEL= ID_FS_LABEL_ENC=
April 30, 200917 yr these lines in the syslog seem to indicate there is some issue with the reiser file-system on disk4, and that is why is did not get mounted: pr 28 20:15:34 Tower kernel: ReiserFS: warning: is_leaf: free space seems wrong: level=1, nr_items=23, free_space=200 rdkey Apr 28 20:15:34 Tower kernel: ReiserFS: md4: warning: vs-5150: search_by_key: invalid format found in block 8211. Fsck? Apr 28 20:15:34 Tower kernel: ReiserFS: md4: warning: vs-13070: reiserfs_read_locked_inode: i/o failure occurred trying to find stat data of [1 2 0x0 SD] Apr 28 20:15:34 Tower kernel: ReiserFS: md4: Using r5 hash to sort names Apr 28 20:15:34 Tower kernel: ReiserFS: md4: warning: xattrs/ACLs enabled and couldn't find/create .reiserfs_priv. Failing mount. Apr 28 20:15:34 Tower kernel: can't shrink filesystem on-line Apr 28 20:15:34 Tower kernel: can't shrink filesystem on-line Apr 28 20:15:34 Tower emhttp: disk4 mount error: 32 The step to perform next is to do a reiserfsck on /dev/md4 since the drive is not mounted, you do not need to un-mount it for the next step. Type reiserfsck -y /dev/md4 Let it run. Odds are it will say it needs to be run again with --fix-fixable as an argument. If it does, do so. Joe L.
April 30, 200917 yr Author I got pages and pages of these, is it safe to rebuild the tree? Tim ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ / 2 (of 123)bad_stat_data: The objectid (301727) is marked free, but used by an object [301723 301727 0x0 SD (0)] / 30 (of 123)bad_stat_data: The objectid (301728) is marked free, but used by an object [301723 301728 0x0 SD (0)] bad_stat_data: The objectid (301731) is marked free, but used by an object [301723 301731 0x0 SD (0)] bad_stat_data: The objectid (301729) is marked free, but used by an object [301728 301729 0x0 SD (0)] bad_stat_data: The objectid (301730) is marked free, but used by an object [301728 301730 0x0 SD (0)] bad_stat_data: The objectid (301732) is marked free, but used by an object [301731 301732 0x0 SD (0)] / 52 (of 123)bad_stat_data: The objectid (301742) is marked free, but used by an object [301741 301742 0x0 SD (0)] bad_stat_data: The objectid (301743) is marked free, but used by an object [301741 301743 0x0 SD (0)] /120 (of 123)bad_stat_data: The objectid (301744) is marked free, but used by an object [301741 301744 0x0 SD (0)] bad_stat_data: The objectid (301746) is marked free, but used by an object [301745 301746 0x0 SD (0)] bad_stat_data: The objectid (301747) is marked free, but used by an object [301745 301747 0x0 SD (0)] /122 (of 123)bad_stat_data: The objectid (301748) is marked free, but used by an object [301745 301748 0x0 SD (0)] bad_stat_data: The objectid (301749) is marked free, but used by an object [301748 301749 0x0 SD (0)] finished Comparing bitmaps..vpf-10640: The on-disk and the correct bitmaps differs. Bad nodes were found, Semantic pass skipped 1 found corruptions can be fixed only when running with --rebuild-tree ########### reiserfsck finished at Wed Apr 29 23:31:15 2009 ########### root@Tower:/#
April 30, 200917 yr I got pages and pages of these, is it safe to rebuild the tree? Tim ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ / 2 (of 123)bad_stat_data: The objectid (301727) is marked free, but used by an object [301723 301727 0x0 SD (0)] / 30 (of 123)bad_stat_data: The objectid (301728) is marked free, but used by an object [301723 301728 0x0 SD (0)] bad_stat_data: The objectid (301731) is marked free, but used by an object [301723 301731 0x0 SD (0)] bad_stat_data: The objectid (301729) is marked free, but used by an object [301728 301729 0x0 SD (0)] bad_stat_data: The objectid (301730) is marked free, but used by an object [301728 301730 0x0 SD (0)] bad_stat_data: The objectid (301732) is marked free, but used by an object [301731 301732 0x0 SD (0)] / 52 (of 123)bad_stat_data: The objectid (301742) is marked free, but used by an object [301741 301742 0x0 SD (0)] bad_stat_data: The objectid (301743) is marked free, but used by an object [301741 301743 0x0 SD (0)] /120 (of 123)bad_stat_data: The objectid (301744) is marked free, but used by an object [301741 301744 0x0 SD (0)] bad_stat_data: The objectid (301746) is marked free, but used by an object [301745 301746 0x0 SD (0)] bad_stat_data: The objectid (301747) is marked free, but used by an object [301745 301747 0x0 SD (0)] /122 (of 123)bad_stat_data: The objectid (301748) is marked free, but used by an object [301745 301748 0x0 SD (0)] bad_stat_data: The objectid (301749) is marked free, but used by an object [301748 301749 0x0 SD (0)] finished Comparing bitmaps..vpf-10640: The on-disk and the correct bitmaps differs. Bad nodes were found, Semantic pass skipped 1 found corruptions can be fixed only when running with --rebuild-tree ########### reiserfsck finished at Wed Apr 29 23:31:15 2009 ########### root@Tower:/# If it says to rebuild tree, then I'd say that is the next step. Just to make you feel better, rebuild tree has been very capable in restoring almost everything in a corrupted file-system even when a user has started to clear it by overwriting it. See this thread for how it worked for that user. So, invoke reiserfsck -y --rebuild-tree /dev/md4 Joe L.
April 30, 200917 yr Author Success! After running the rebuild tree, I rechecked the vol_id and still didn’t see anything: root@Tower:/dev# vol_id /dev/sdd1 /dev/sdd1: unknown volume type I then stopped the array and restarted it. I noticed a bunch of writes to both the parity disk and to disk4. I got that sinking feeling in my stomach but after a minute, I hit refresh and unraid showed the disk as formatted and mounted! It was online as 1.5tb, having about the same free space as before the crash plus the additional 500gb. I went to the disk and noticed only a lost+found directory. Going in random directory name after name, I saw all of my old files. I started opening them up and from what I checked, all looks good! All I need to do is rename a couple directories and I’m back up and running . I’ll press “check” tonight, from what I gather, that’s how I do a rebuild of my parity disk from the recovered data. At least for me, I would think it would force a check of your data disks, not a regeneration of parity, but I’ve learned that not is all as it seems with those buttons! I hope that if any others experience what I did, read the posts in order, carefully and go slow. Don’t be afraid to ask questions here, I’ve learned a lot from this experience and people here are more than will to help and teach you. The meat of the recovery took place by doing this (exactly!): http://lime-technology.com/forum/index.php?topic=3716.msg32292#msg32292 A huge, huge thanks goes out to Joe L, bjp99 and lionelhutz. Without there help and willingness to spend time with me, I would be left with a large data loss which was the exact reason why I invested a lot of money into this system in the first place. I really can’t thank you all enough!!! Tim
April 30, 200917 yr Success! After running the rebuild tree, I rechecked the vol_id and still didn’t see anything: root@Tower:/dev# vol_id /dev/sdd1 /dev/sdd1: unknown volume type I then stopped the array and restarted it. I noticed a bunch of writes to both the parity disk and to disk4. I got that sinking feeling in my stomach but after a minute, I hit refresh and unraid showed the disk as formatted and mounted! It was online as 1.5tb, having about the same free space as before the crash plus the additional 500gb. I went to the disk and noticed only a lost+found directory. Going in random directory name after name, I saw all of my old files. I started opening them up and from what I checked, all looks good! All I need to do is rename a couple directories and I’m back up and running . That is what was expected. Yes, you will have to rename some folders. I’ll press “check” tonight, from what I gather, that’s how I do a rebuild of my parity disk from the recovered data. At least for me, I would think it would force a check of your data disks, not a regeneration of parity, but I’ve learned that not is all as it seems with those buttons! Correct. Because you did the file-system repair on the /dev/md4 device, the parity drive was updated as the repair was performed. Now, you might still have a few parity errors that will get corrected when you press the "Check" button, but they will most likely be near the beginning of the disks and in the housekeeping areas of the disks. If you do get a few errors, just run another parity check... it should come out clean. I hope that if any others experience what I did, read the posts in order, carefully and go slow. Don’t be afraid to ask questions here, I’ve learned a lot from this experience and people here are more than will to help and teach you. The meat of the recovery took place by doing this (exactly!): http://lime-technology.com/forum/index.php?topic=3716.msg32292#msg32292 A huge, huge thanks goes out to Joe L, bjp99 and lionelhutz. Without there help and willingness to spend time with me, I would be left with a large data loss which was the exact reason why I invested a lot of money into this system in the first place. I really can’t thank you all enough!!! Tim You are welcome. Also, once you are back running normally, take a few moments to make a copy of your config folder with the current configuration and the array in a stopped state. 1. Stop the array 2. From a linux prompt type (using the current date) cp -r /boot/config /boot/config2009-04-30 3. Then re-start the array from the management console. If you ever run into issues, we have lots more ways to help get back to normal if you had saved a valid copy of the files in the config folder when all was running properly and the array stopped cleanly. 4. Each time you add/replace/ or delete a drive from the array, make another copy of the config folder named after the then current date. Joe L.
May 1, 200917 yr That's great to hear. Sounds like a few previous cases. Everything was there but directory and file names get messed up. Joe is the man when it comes to recovering an unRAID drive after all appears lost. Peter
Archived
This topic is now archived and is closed to further replies.