June 14, 201016 yr What I'm going to suggest is if we do not have enough memory to fix the file-system we simply re-format the drive and THEN run the command to rebuild the file-tree scanning the entire drive. I'm going to try it here on a drive not in my array first to see how it goes. I've got an old 8Gig drive in my array I keep for just such tests. i'll create a file-system on it, fill it with files and directories, re-format it, and then run the --rebuild-tree scanning the entire disk. I can then see what it was able to recover. This should not take too long since it is a small disk. Joe L. Ok, so if you wanted to try that; interesting thought. Being a memory issue though, I can try throwing more memory at it. Right now I have 2x 256MB sticks in there but if I find say 2x 512MB, I wonder if that would be enough? (That's likely all I will be able to find). Obviously that will mean shutting down the system. Maybe I'll capture the syslog before doing so for reference. Anything else I should do? Once I have the new mem in, should I start from the top with everything except the rebuild-sb? yes, you are running out of memory to run the process. Yes, add the memory strips. don't worry about the syslog. I believe you have discovered why the reiserfsck cannot complete. We'll soon see if 1Gig is enough. If you have the capability, perhaps you can use both sets of strips in your server at the same time. (if you have 4 slots for RAM) Don't do the rebuild-sb. Just do the rebuild-tree. reiserfsck --rebuild-tree /dev/md1 Once it is sane you can try: reiserfsck --rebuild-tree --scan-whole-partition /dev/md1 Joe L.
June 14, 201016 yr Author Bah, it's a P4 system so I thought for sure it was using Rambus memory which I was hoping to borrow from work. Turns out it uses DDR2 which I have plenty of at home. This is good and bad; good because I have 2GB I can throw at it but bad because I can't touch it for another 5 hours. Will update tonight after re-running those commands.
June 15, 201016 yr Author Ok, I pulled 2GB of memory out of my other server and started the process again: 1) Installed the latest version of reiserfsprogs (3.6.21) 2) Stopped Samba & unmounted the fs 3) Ran reiserfsck --check /dev/md1 and got: Do you want to run this program?[N/Yes] (note need to type Yes if you do):Yes ########### reiserfsck --check started at Mon Jun 14 17:47:41 2010 ########### Replaying journal: Done. Reiserfs journal '/dev/md1' in blocks [18..8211]: 0 transactions replayed Zero bit found in on-disk bitmap after the last valid bit. Checking internal tree.. Bad root block 0. (--rebuild-tree did not complete) Aborted 4) Ran reiserfsck --fix-fixable /dev/md1 Do you want to run this program?[N/Yes] (note need to type Yes if you do):Yes ########### reiserfsck --fix-fixable started at Mon Jun 14 17:52:02 2010 ########### Replaying journal: Done. Reiserfs journal '/dev/md1' in blocks [18..8211]: 0 transactions replayed Zero bit found in on-disk bitmap after the last valid bit. Fixed. Checking internal tree.. Bad root block 0. (--rebuild-tree did not complete) Aborted 5) Ran reiserfsck --rebuild-tree /dev/md1 Do you want to run this program?[N/Yes] (note need to type Yes if you do):Yes Replaying journal: Done. Reiserfs journal '/dev/md1' in blocks [18..8211]: 0 transactions replayed Zero bit found in on-disk bitmap after the last valid bit. Fixed. ########### reiserfsck --rebuild-tree started at Mon Jun 14 18:00:25 2010 ########### Pass 0: ####### Pass 0 ####### Loading on-disk bitmap .. ok, 79130755 blocks marked used Skipping 19389 blocks (super block, journal, bitmaps) 79111366 blocks will be read 0%block 10689002: The number of items ( is incorrect, should be (0) - corrected block 10689002: The free space (7) is incorrect, should be (4072) - corrected .block 21949326: The number of items (1) is incorrect, should be (0) - corrected block 21949326: The free space ( is incorrect, should be (4072) - corrected block 21950313: The number of items (1) is incorrect, should be (0) - corrected block 21950313: The free space ( is incorrect, should be (4072) - corrected ...20%.block 107354135: The number of items (1) is incorrect, should be (0) - corrected block 107354135: The free space ( is incorrect, should be (4072) - corrected block 107355122: The number of items (1) is incorrect, should be (0) - corrected block 107355122: The free space ( is incorrect, should be (4072) - corrected .block 138667788: The number of items (2) is incorrect, should be (0) - corrected block 138667788: The free space (12) is incorrect, should be (4072) - corrected .block 144050718: The number of items (2) is incorrect, should be (0) - corrected block 144050718: The free space (12) is incorrect, should be (4072) - corrected .block 166769764: The number of items (3328) is incorrect, should be (1) - corrected block 166769764: The free space (768) is incorrect, should be (720) - corrected pass0: vpf-10110: block 166769764, item (0): Unknown item type found [3204449536 469762129 0x0 (15)] - deleted 40%..block 210881533: The free space (0) is incorrect, should be (4048) - corrected pass0: vpf-10110: block 210881533, item (0): Unknown item type found [1208877058 3820628 0x690072 (15)] - deleted .block 216625319: The number of items (6679) is incorrect, should be (1) - corrected block 216625319: The free space (6696) is incorrect, should be (4048) - corrected pass0: vpf-10200: block 216625319, item 0: The item [439162081 64422870 0xa2f00011a1d1a2e IND (1)] with wrong offset is deleted block 217345312: The number of items (8392) is incorrect, should be (1) - corrected block 217345312: The free space (8406) is incorrect, should be (4048) - corrected pass0: vpf-10110: block 217345312, item (0): Unknown item type found [551231668 81928412 0x20ca20c2 (15)] - deleted block 347410342: The number of items (256) is incorrect, should be (1) - corrected block 347410342: The free space (256) is incorrect, should be (4048) - corrected pass0: vpf-10110: block 347410342, item (0): Unknown item type found [131075 1075052544 0x4f400000000006 (4)] - deleted left 0, 19902 /secc 136322 directory entries were hashed with "r5" hash. "r5" hash is selected Flushing..finished Read blocks (but not data blocks) 79111366 Leaves among those 97712 - leaves all contents of which could not be saved and deleted 18 Objectids found 136324 Pass 1 (will try to insert 97694 leaves): ####### Pass 1 ####### Looking for allocable blocks .. finished 0%....20%....40%....60%....80%....100% left 0, 159 /sec Flushing..finished 97694 leaves read 97371 inserted 323 not inserted ####### Pass 2 ####### Pass 2: 0%....20%....40%....60%....80%....100% left 0, 161 /sec Flushing..finished Leaves inserted item by item 323 Pass 3 (semantic): ####### Pass 3 ######### Flushing..finished Files found: 124612 Directories found: 11712 Pass 3a (looking for lost dir/files): ####### Pass 3a (lost+found pass) ######### Looking for lost directories: Flushing..finished0, 207 /sec Pass 4 - finished done 7414, 31 /sec Flushing..finished Syncing..finished ########### reiserfsck finished at Mon Jun 14 19:24:09 2010 ########### 6) Now I'm going to run reiserfsck --rebuild-tree --scan-whole-partition /dev/md1
June 15, 201016 yr 6) Now I'm going to run reiserfsck --rebuild-tree --scan-whole-partition /dev/md1 great news.
June 15, 201016 yr Author great news. Yes, I especially like this part: Files found: 124612 Directories found: 11712 So come morning, when this is (hopefully, successfully) done, my next step would be to... (I'm guessing) mount /dev/md1 /mnt/disk1 /usr/sbin/smbd -D /usr/sbin/nmbd -D and see what I can see?
June 15, 201016 yr Author Ok well... it finished! Ended with: Flushing..finished Objects without names 385 Empty lost dirs removed 2 Dirs linked to /lost+found: 14 Dirs without stat data found 1 Files linked to /lost+found 371 Objects having used objectids: 220 files fixed 207 dirs fixed 13 Pass 4 - finished done 45713, 123 /sec Deleted unreachable items 1506 Flushing..finished Syncing..finished ########### reiserfsck finished at Tue Jun 15 00:47:21 2010 ########### Now when I try and mount the fs I get "mount point mnt/disk1 does not exist. I found one other reference to this on the forum indicating I can create one again or just reboot. Which would you suggest for me?
June 15, 201016 yr probably easier to just reboot, that way the management console will be in sync and samba started.
June 15, 201016 yr Author It's... not showing up as unformatted anymore... I'm seeing both the original share & lost+found... (heart beating, could it be?) There is over 5GB in lost and found. The rest of the original directory structure appears to exist! It's going to take some time to sort through this all but your efforts deserve a big THANK YOU right now! Could NOT have done this without your help Joe. I truly appreciate it. Now let's just hope most wasn't lost in the shuffle Edit: It does appear that there are a fair amount of files missing/unable to open however it'll still take time to see if they exist in lost+found etc. I'll have to go back to those threads indicated in the wiki for sorting through all of this. What I have right now though is significantly more than what I've had access to since Thursday!
June 20, 201016 yr Author Joe, thanks again for all your help. After sifting through gigabytes of data, I've determined a smallish (still significant) amount missing so my last question to you is, what can I do to use the 2TB drive in an unraid box? You've told me how to copy it back to the 1.5tb drive but if possible, I'd rather start pre-clearing my 1.5tb drives and get the server back up and running with them. Can I use the 2TB drive on its own? Thanks.
June 21, 201016 yr Joe, thanks again for all your help. After sifting through gigabytes of data, I've determined a smallish (still significant) amount missing so my last question to you is, what can I do to use the 2TB drive in an unraid box? You've told me how to copy it back to the 1.5tb drive but if possible, I'd rather start pre-clearing my 1.5tb drives and get the server back up and running with them. Can I use the 2TB drive on its own? Thanks. To use the 2TB drive all you need to do is set the partition type in the MBR. Right now it is still set to zero from the pre-clearing process. Easiest way to do that is to type: echo -ne "\203" | dd bs=1 count=1 seek=450 of=/dev/sdX where sdX = the correct drive device. I think the 2TB drive was /dev/sda. Note: this time we DO NOT want to write to the first partition, so we do NOT put a "1" on the end of the device name.
June 21, 201016 yr Author Great, thanks Joe. I have been using TeraCopy to transfer most of my data back (& do CRC checks) and have noticed that sometimes it will give "cannot open file" errors (among others), leading me to believe files are corrupted. Sometimes they are but I've found that many can be copied over in smaller chunks (fewer files/folders at a time) so it looks like I'm getting quite a good amount back. Either tonight or tomorrow I'll re-start building the server again. I just wanted to make sure if I need to try to dig for anything again that I could do it directly from the 2TB drive which it appears is the case.
June 23, 201016 yr Author So I was running the preclear script on both 1.5TB drives once again before setting everything up again. I did so via telnet and when checking before I went to bed the 10 of 10 processes were listed and it was working on its final read sequence. At say 10:30pm it was about 6% done, reading at just under 100MB/s. Being Tuesday Microsoft decided to push out updates which resulted in my computer restarting sometime after 3am. Needless to say, that terminated the telnet sessions. I read elsewhere on the forums you said the process would have to be restarted. a) do you think it might have had time to finish by then b) even if it didn't, do I really have to lose another day re-running the script? (I had started it around 10:30am the day previous.) What's the worst that would happen?
June 23, 201016 yr So I was running the preclear script on both 1.5TB drives once again before setting everything up again. I did so via telnet and when checking before I went to bed the 10 of 10 processes were listed and it was working on its final read sequence. At say 10:30pm it was about 6% done, reading at just under 100MB/s. Being Tuesday Microsoft decided to push out updates which resulted in my computer restarting sometime after 3am. Needless to say, that terminated the telnet sessions. I read elsewhere on the forums you said the process would have to be restarted. a) do you think it might have had time to finish by then b) even if it didn't, do I really have to lose another day re-running the script? (I had started it around 10:30am the day previous.) What's the worst that would happen? You could look in the syslog. The final output is captured there if they did complete. If it was reading at 100MB/s, then it was taking 10 seconds per GB. To read the remaining 1410 GB it would have taken roughly 4 hours more. There is a possibility they completed. Regardless, you can get a final smart report on each drive and look for re-allocated sectors and sectors pending re-allocation. smartctl -d ata -a /dev/sdX You can also run preclear_disk.sh -t /dev/sdX to test if the pre-clear signature is present.
June 23, 201016 yr Author From what I can tell, re-allocated sectors and those pending are both showing 0 on each drive: sda SMART Attributes Data Structure revision number: 16 Vendor Specific SMART Attributes with Thresholds: ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 1 Raw_Read_Error_Rate 0x000f 100 100 051 Pre-fail Always - 0 3 Spin_Up_Time 0x0007 071 071 011 Pre-fail Always - 9510 4 Start_Stop_Count 0x0032 100 100 000 Old_age Always - 118 5 Reallocated_Sector_Ct 0x0033 100 100 010 Pre-fail Always - 0 7 Seek_Error_Rate 0x000f 253 253 051 Pre-fail Always - 0 8 Seek_Time_Performance 0x0025 100 100 015 Pre-fail Offline - 0 9 Power_On_Hours 0x0032 100 100 000 Old_age Always - 1868 10 Spin_Retry_Count 0x0033 100 100 051 Pre-fail Always - 0 11 Calibration_Retry_Count 0x0012 100 100 000 Old_age Always - 0 12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 22 13 Read_Soft_Error_Rate 0x000e 100 100 000 Old_age Always - 0 183 Unknown_Attribute 0x0032 100 100 000 Old_age Always - 0 184 Unknown_Attribute 0x0033 100 100 000 Pre-fail Always - 0 187 Reported_Uncorrect 0x0032 100 100 000 Old_age Always - 0 188 Unknown_Attribute 0x0032 100 100 000 Old_age Always - 0 190 Airflow_Temperature_Cel 0x0022 077 064 000 Old_age Always - 23 (Lifetime Min/Max 23/23) 194 Temperature_Celsius 0x0022 075 063 000 Old_age Always - 25 (Lifetime Min/Max 23/25) 195 Hardware_ECC_Recovered 0x001a 100 100 000 Old_age Always - 1104 196 Reallocated_Event_Count 0x0032 100 100 000 Old_age Always - 0 197 Current_Pending_Sector 0x0012 100 100 000 Old_age Always - 0 198 Offline_Uncorrectable 0x0030 100 100 000 Old_age Offline - 0 199 UDMA_CRC_Error_Count 0x003e 100 100 000 Old_age Always - 0 200 Multi_Zone_Error_Rate 0x000a 100 100 000 Old_age Always - 0 201 Soft_Read_Error_Rate 0x000a 253 253 000 Old_age Always - 0 sdb SMART Attributes Data Structure revision number: 16 Vendor Specific SMART Attributes with Thresholds: ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 1 Raw_Read_Error_Rate 0x000f 100 100 051 Pre-fail Always - 0 3 Spin_Up_Time 0x0007 072 072 011 Pre-fail Always - 9330 4 Start_Stop_Count 0x0032 100 100 000 Old_age Always - 135 5 Reallocated_Sector_Ct 0x0033 100 100 010 Pre-fail Always - 0 7 Seek_Error_Rate 0x000f 253 253 051 Pre-fail Always - 0 8 Seek_Time_Performance 0x0025 100 100 015 Pre-fail Offline - 13070 9 Power_On_Hours 0x0032 100 100 000 Old_age Always - 2020 10 Spin_Retry_Count 0x0033 100 100 051 Pre-fail Always - 0 11 Calibration_Retry_Count 0x0012 100 100 000 Old_age Always - 0 12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 37 13 Read_Soft_Error_Rate 0x000e 100 100 000 Old_age Always - 0 183 Unknown_Attribute 0x0032 100 100 000 Old_age Always - 0 184 Unknown_Attribute 0x0033 100 100 000 Pre-fail Always - 0 187 Reported_Uncorrect 0x0032 100 100 000 Old_age Always - 0 188 Unknown_Attribute 0x0032 100 100 000 Old_age Always - 0 190 Airflow_Temperature_Cel 0x0022 075 060 000 Old_age Always - 25 (Lifetime Min/Max 25/25) 194 Temperature_Celsius 0x0022 073 059 000 Old_age Always - 27 (Lifetime Min/Max 25/27) 195 Hardware_ECC_Recovered 0x001a 100 100 000 Old_age Always - 988 196 Reallocated_Event_Count 0x0032 100 100 000 Old_age Always - 0 197 Current_Pending_Sector 0x0012 100 100 000 Old_age Always - 0 198 Offline_Uncorrectable 0x0030 100 100 000 Old_age Offline - 0 199 UDMA_CRC_Error_Count 0x003e 100 100 000 Old_age Always - 0 200 Multi_Zone_Error_Rate 0x000a 100 100 000 Old_age Always - 0 201 Soft_Read_Error_Rate 0x000a 253 253 000 Old_age Always - 0 As far as the preclear check is concerned, looks like I'm good to go! Pre-Clear unRAID Disk ######################################################################## Device Model: SAMSUNG HD154UI Serial Number: S1XWJ1MZ205901 Firmware Version: 1AG01118 User Capacity: 1,500,301,910,016 bytes Disk /dev/sda: 1500.3 GB, 1500301910016 bytes 255 heads, 63 sectors/track, 182401 cylinders, total 2930277168 sectors Units = sectors of 1 * 512 = 512 bytes Disk identifier: 0x00000000 Device Boot Start End Blocks Id System /dev/sda1 63 2930277167 1465138552+ 0 Empty Partition 1 does not end on cylinder boundary. ######################################################################## ============================================================================ == == DISK /dev/sda IS PRECLEARED == and Pre-Clear unRAID Disk ######################################################################## Device Model: SAMSUNG HD154UI Serial Number: S1XWJ1MZ205904 Firmware Version: 1AG01118 User Capacity: 1,500,301,910,016 bytes Disk /dev/sdb: 1500.3 GB, 1500301910016 bytes 255 heads, 63 sectors/track, 182401 cylinders, total 2930277168 sectors Units = sectors of 1 * 512 = 512 bytes Disk identifier: 0x00000000 Device Boot Start End Blocks Id System /dev/sdb1 63 2930277167 1465138552+ 0 Empty Partition 1 does not end on cylinder boundary. ######################################################################## ============================================================================ == == DISK /dev/sdb IS PRECLEARED == Thanks once again! Almost back on my feet...
July 7, 201016 yr Author I could use a bit more guidance Joe (or anyone else). Of everything, there are 3-5 rather important files that I was not able to recover. They are chapters of a book currently being authored. I have print-outs of these chapters but not being able to access several of them means having to re-create them manually. I'm going to try and work with the 2TB drive to see if I have any better luck recovering them. Thing is, I'm not off to a great start. I went back using a "new" usb key that had 4.5.4 installed, and only have the 2TB drive attached currently. You previously gave directions on how to set the partition type so I could use the drive. I did that, it said 1k was written but after installing the newer version of reiserfsprogs and trying to run reiserfs --check I get Failed to open the device '/dev/md1': No such file or directory I'm not clear on whether I should have attempted that with the array started, with samba stopped or ... When I do start the array, the disk is showing up as unformatted. I obviously have to get the drive properly recognized before I can attempt recovery as before.
Archived
This topic is now archived and is closed to further replies.