January 25, 201016 yr I logged in to my unraid through my browser, and one of my hard drives had a red arrow, so it was disabled, with some write errors beside it. In all my wisdom, the "restore" option seemed the best option, my server crashed during this operation (which I now know I should not have done) and when I started back up I unassigned the drive, rebooted, re-assigned the drive and started the parity sync / re-build of the drive. 5 minutes into this, I realized my drive no longer had approximately 6 months of data on it! I cancelled the re-build about 25 gigs in, out of a drive that used to have 900 gigs on it, but now was only reporting 600 gigs. So I guess i have restored the drive to a time in the past, and have lost 300 gigs of data. This data represents thousands of hours of work, my business files, work files, etc. All gone. Is there ANY way to recover what I have lost from this drive? The most important data I have lost is only a few hudred megs, maybe a couple of gigs. Can I pay any companies or persons to recover this data? I am desperate, I hope somebody knows a way to recover my data, please.
January 25, 201016 yr I logged in to my unraid through my browser, and one of my hard drives had a red arrow, so it was disabled, with some write errors beside it. In all my wisdom, the "restore" option seemed the best option, my server crashed during this operation The button labeled "restore" has absolutely nothing to do with restoring data of a failed disk to its replacement. It is very poorly labeled, as it is actually a button that immediately invalidates parity and sets a brand new configuration based on the currently working and assigned disks. Using it when a disk has been taken off-line because of write failures will eliminate almost any chance of rebuilding its old contents. Describe more how your server "crashed" during the initial "restore." If the drive had actually failed this "restore" would start writing parity based on the remaining working disks. If it was working at that time it would have just invalidated parity and started the process of writing it again. I'm trying to figure out just what happened at that point. Did it look like the disk was active? Did it ever finish this parity calc? Or did it crash part way through? (which I now know I should not have done) and when I started back up I unassigned the drive, rebooted, re-assigned the drive and started the parity sync / re-build of the drive. If the parity calc had completed without the failed drive, then it is not known at all. If it did not complete, then it is a complete mystery on what is in parity Assuming you convinced the server that parity was OK then un-assigning, rebooting, and re-assigning the defective drive would start the process of re-constructing what the array thought was on it based on parity and the remaining data drives. 5 minutes into this, I realized my drive no longer had approximately 6 months of data on it! I cancelled the re-build about 25 gigs in, out of a drive that used to have 900 gigs on it, but now was only reporting 600 gigs. It sounds as if your parity was corrupted when you requested it to forget its old contents and then partially saved new parity to it without the defective drive. So I guess i have restored the drive to a time in the past, and have lost 300 gigs of data. It does not work that way. Unless you have saved incremental backups of your data, there is no way to "restore" to a point of time in the past... Your usage of the term "restore" is the more like how I use it... but do not confuse that process with the button labeled "restore" It is a "Set New Disk Configuration" button. I feel your pain... This data represents thousands of hours of work, my business files, work files, etc. All gone. Is there ANY way to recover what I have lost from this drive? The most important data I have lost is only a few hudred megs, maybe a couple of gigs. First un-assign the disk from the array. We do NOT want it writing to the disk. Next... purchase a replacement disk. We want to copy whatever we can to a safe place and you probably have to deal with whatever took the disk off-line in the first place. Your best bet at recovering your files is from the disk you partially restored to. You've probably "lost" files because you've clobbered the file-system in attempting to rebuild it. Can I pay any companies or persons to recover this data? I am desperate, I hope somebody knows a way to recover my data, please. The basic plan is this: Copy from the failed drive to a replacement. (If possible) If the failed drive is un-readable... goto plan "B" Use a file-system repair tool on the replacement drive. Once sane, use rebuild-tree on the entire drive to attempt to recover what you can. Plan "B" Send the failed drive to a data recovery service. Yes, there are people/companies who might be able to recover some data from the drive. They will not be able to recover anything you overwrote. It will not be inexpensive. Oh yes... one last thing, when I was looking back on your older posts to learn more about your system I found this paragraph written by you (perhaps that is plan "C" if plan "A" and "B" don't work): And all my critical files are backed up on DVD and off-line computers anyways. If I get hacked then so what? I load my saved unRaid config onto my USB and start over. Joe L.
January 25, 201016 yr Author Joe, I appreciate your quick and thorough response, thank you for trying to help. I will try to answer your questions below: 1. When I clicked on "restore" then I left the computer for a few hours, came back to check on it, and I could not access UnRaid through my browser. I had to power down the server, and reboot. The array was online when I booted, but the same drive was still disabled. I can't remember at this time (didn't look) to see if the amount of "free" space on the drive represented the 3-400 gigs of lost data or not. I did some more reading, then unassigned, rebooted, and assigned the disk again, and started the re-build. 25 gigs in to the rebuild I realized my error using "restore" and cancelled it. The drive is now unassigned from my array. I am confident that my disk drive still functions, as it started to re-build it (got about 25 gigs worth without any write errors). Your Plan A: "Copy from the failed drive to a replacement. Use a file-system repair tool on the replacement drive. Once sane, use rebuild-tree on the entire drive to attempt to recover what you can." How do I start this process once I have a replacement disk? Do I do this within unRaid, or in Windows or something else? Out of the 3-400 gigs of data "lost" from the parity/drive information, I only really NEED a couple of gigs of business files, so hopefully with a recovery program I can pull that data from the drive somehow, and all is not lost. The majority is just music, movies, etc that I don't care about, but my business files (I am an independent consultant) are priceless to me. Unforunately plan C (extrenal backups) will recover some of my oldest data, but I stopped DVD backups once I was comfortable unRaid was functioning well, I actually had a disk fail and it rebuilt it perfectly, so I used unRaid as my only back from then on. Guess I should add back in my external backups!
January 25, 201016 yr Joe, I appreciate your quick and thorough response, thank you for trying to help. I will try to answer your questions below: 1. When I clicked on "restore" then I left the computer for a few hours, came back to check on it, and I could not access UnRaid through my browser. I had to power down the server, and reboot. The array was online when I booted, but the same drive was still disabled. I can't remember at this time (didn't look) to see if the amount of "free" space on the drive represented the 3-400 gigs of lost data or not. I did some more reading, then unassigned, rebooted, and assigned the disk again, and started the re-build. 25 gigs in to the rebuild I realized my error using "restore" and cancelled it. The drive is now unassigned from my array. Basically, we have no way to know how far it proceeded before the failing drive was taken off line. We can guess the errors filled the syslog and eventually crashed the server as it killed processes to free ram. I am confident that my disk drive still functions, as it started to re-build it (got about 25 gigs worth without any write errors). We know it works for a short time. We do not know much more about it. You can run a SMART report on it smartctl -a -d ata /dev/sdX where sdX = the device of the disk. (Seen in parens on the "devices" page) Your Plan A: "Copy from the failed drive to a replacement. Use a file-system repair tool on the replacement drive. Once sane, use rebuild-tree on the entire drive to attempt to recover what you can." How do I start this process once I have a replacement disk? Do I do this within unRaid, or in Windows or something else? You can do it on the unRAID server, but not through its web-interface. It is a command line process. How big is the disk that you clobbered? Do you have a spare disk of that size? or bigger? If not, now would not be the time to wait for a sale. Purchase one. Install it on a spare port in your server. DO NOT assign it to the unRAID array. Leave the disk you clobbered un-assigned. we do not want to write to it in any way. We will use the "dd" command to copy it to the new disk you install. That will give you two identical corrupted copies of the disk you clobbered. We can then attempt recovery on the copied one without risking anything, since the original clobbered data will always be available on the original disk. Out of the 3-400 gigs of data "lost" from the parity/drive information, I only really NEED a couple of gigs of business files, so hopefully with a recovery program I can pull that data from the drive somehow, and all is not lost. The majority is just music, movies, etc that I don't care about, but my business files (I am an independent consultant) are priceless to me. Unfortunately, you do not get to choose what to recover. If the files you need are not in the section of the disk you overwrote, there is some hope. Unforunately plan C (extrenal backups) will recover some of my oldest data, but I stopped DVD backups once I was comfortable unRaid was functioning well, I actually had a disk fail and it rebuilt it perfectly, so I used unRaid as my only back from then on. Guess I should add back in my external backups! A RAID array is not a substitute for backups to external media. All it takes is a single fire/flood/tornado/lightning strike/multi-disk-oops and your data is toast. It would never protect yourself from an errant deltree *.*. Best is if the copies are off-site. Joe L.
January 25, 201016 yr Author I will buy a new disk the same size tomorrow morning, it is a 1.5 Terabyte, I don't have anything that large spare. 1. I am not a Linux-savvy person though, can you provide step-by-step for copying the clobbered drive contents to the new one? Once the drive is copied, how do I attempt recovery? 2. Is there anything I can do in the meantime, is it possible to put the disk into a windows box and read the data from it? That way I will know whether my important data still exists or not?
January 25, 201016 yr I will buy a new disk the same size tomorrow morning, it is a 1.5 Terabyte, I don't have anything that large spare. 1. I am not a Linux-savvy person though, can you provide step-by-step for copying the clobbered drive contents to the new one? Yes... Once the drive is copied, how do I attempt recovery? We'll use reiserfsck on the copy of the data. 2. Is there anything I can do in the meantime, is it possible to put the disk into a windows box and read the data from it? That way I will know whether my important data still exists or not? You can mount the drive as read-only outside of the array and browse it using "mc" If the file-system is corrupted there is nothing you can do on windows to fix it. We'll need to run a file-system-check on it to check for corruption, and it is almost certainly corrupted. How much "data" is on the failed drive? I think you mentioned 900 Meg. Do you know the linux device of the drive? sda, sdb, sdc??? Joe L.
January 25, 201016 yr Author Yes, it is SDC I know how to login via telnet, but thats about it. What commands would I use to mount the drive?
January 25, 201016 yr Author The drive had about 6-700 gigs on it before, now unraid is showing about 300 gigs on it. The missing 2-300 gigs is my files for the past year or so. I can't buy the new drive until tomorrow, and I really want to know whether my critical business files still exist or not ASAP, any way to check the drive tonight for corruption? Thank you Joe, Rory
January 25, 201016 yr Yes, it is SDC I know how to login via telnet, but thats about it. What commands would I use to mount the drive? To mount the drive we first need a mount point. (An empty directory on which to mount the disk) You can simply make one mkdir /tmp/bad_disk Then to mount the disk on that mount point (note the "1" on the end of the device name to indicate the first partition on the disk) mount -r -t reiserfs /dev/sdc1 /tmp/bad_disk Then you can invoke "mc" and browse to /tmp/bad_disk When done type umount /tmp/bad_disk Joe L.
January 25, 201016 yr The drive had about 6-700 gigs on it before, now unraid is showing about 300 gigs on it. The missing 2-300 gigs is my files for the past year or so. I can't buy the new drive until tomorrow, and I really want to know whether my critical business files still exist or not ASAP, any way to check the drive tonight for corruption? Thank you Joe, Rory If you still have the disk un-assigned, then what you are seeing through the LAN is the simulated disk created from parity and the other data disks. If your files are there, great. If not, then mounting the physical disk might let you see the actual files. To run a file system check on the un-assigned disk run reiserfsck /dev/sdc1 (Again, use sdc1 to signify the first partition) If it says it is clean, then the mount would probably work. If the mount did not succeed, the file-system check will probably show corruption. I would not use any of the "fix" options of the file-system check program until you make a copy of the failed disk. Some of the fix actions cannot be un-done and might cause more damage... So, playing it safe at this point is better than you accidentally clobbering the data you are trying to recover. Joe L.
January 25, 201016 yr Author Ok, I mounted it, and browsed to it. There are files there, but not my missing files, it shows the same files as what UnRaid is reporting through my shares (through the simulated disk created from parity) I don't know what is going on. Where did my files go then? Could they have been on a different drive? I am assuming they were on this one, because it has the most "free" room, but they are not there! For example, I had a directory on my share called "Opticor". This is just gone, no such directory any more, or the files within it. Other directories exist, but are empty. Older directores are there, and accessible. So confused, what could have happened? Ok, so there are files on the drive, and mounting it will probably work, but that won't bring my files back!?
January 25, 201016 yr Ok, I mounted it, and browsed to it. There are files there, but not my missing files, it shows the same files as what UnRaid is reporting through my shares (through the simulated disk created from parity) It sounds like your first 25 or so gig you overwrote had the directory pointers for the directory you are missing. The data might be above the point overwritten, or below it. I don't know what is going on. Where did my files go then? Could they have been on a different drive? I am assuming they were on this one, because it has the most "free" room, but they are not there! The equivalent would be if I took the card-catalog out of a library. The books would all still be there, but there would be no way to locate them. For example, I had a directory on my share called "Opticor". This is just gone, no such directory any more, or the files within it. Other directories exist, but are empty. Older directores are there, and accessible. So confused, what could have happened? Ok, so there are files on the drive, and mounting it will probably work, but that won't bring my files back!? un-mount the drive and run the file-system-check on it. Let's see if it says there is corruption. The un-mount command is "umount /tmp/bad_disk" Joe L.
January 25, 201016 yr Could they have been on a different drive? I am assuming they were on this one, because it has the most "free" room, but they are not there! This is exactly the reason why I do not use user-shares when writing critical data to the array. I want to know where it is physically located in case I need to deal with physical disks. I have 10 data disks, and a portion of my movie collection across them in addition to backups of other PCs and personal documents and data. Even the movies are organized by alpha range on the disks. If I were to lose disk1 I'd know it had movies starting with 0-9 and A-B only. disk2 has the next alpha range. etc... If I lose a drive, I know which movies I'll need to re-rip.
January 25, 201016 yr Crazy. Ok the process is running, when it finishes what do I do? You typed reiserfsck /dev/sdc1 right? And then answered "Yes" at the prompt? If so, it will run for as long as 30 minutes or so. When done it will report what it finds. Post its output. DO NOT use any of the "fix" options it might suggest... not yet... not until we have a safe copy of the raw bits on the disk. Even if there are not directory entries, sometimes a company that specializes in recovery can get to the files not having directory entries, and you do not want to clobber their chances. Joe L.
January 25, 201016 yr Ok the process finished, a bunch of text, how can I paste it in here? If you are using telnet, I'd say copy and paste. If using the system console. can't so that so re-run using telnet or paraphrase. Does it say there are errors?? Joe L.
January 25, 201016 yr Author I am using telnet through cmd in Windows 7. I dont see how to copy and paste! Yes, there are errors, the last few lines of text (after a bunch of individual errors): Finished Comparing bitmaps..vpf-10640: The on-disk and the correct bitmaps differs. Bad nodes were found, Semantic pass skipped 8 found corruptions can be fixed only when running with --rebuild-tree Once I get my new drive installed, what are the commands to copy the contents to it? And then once the contents are copied, what are the commands to attempt a repair on the new drive copy? (in case you aren't online in the morning I would like to be ready)
January 25, 201016 yr Author Ok I have copied all of the important data that was left from the UnRaid image onto other drives within the array. I'm not going to send this drive in for repair, so lets do it! What commands do I run to see what repair-tree can do?
January 25, 201016 yr Author Figured out the copy-paste: root@Tower:~# reiserfsck /dev/sdc1 reiserfsck 3.6.19 (2003 www.namesys.com) ************************************************************* ** If you are using the latest reiserfsprogs and it fails ** ** please email bug reports to [email protected], ** ** providing as much information as possible -- your ** ** hardware, kernel, patches, settings, all reiserfsck ** ** messages (including version), the reiserfsck logfile, ** ** check the syslog file for any related information. ** ** If you would like advice on using this program, support ** ** is available for $25 at www.namesys.com/support.html. ** ************************************************************* Will read-only check consistency of the filesystem on /dev/sdc1 Will put log info to 'stdout' Do you want to run this program?[N/Yes] (note need to type Yes if you do):Yes ########### reiserfsck --check started at Sun Jan 24 14:54:40 2010 ########### Replaying journal.. Reiserfs journal '/dev/sdc1' in blocks [18..8211]: 0 transactions replayed Checking internal tree../ 1 (of 5)/ 1 (of 89)/ 93 (of 170)bad_path: block 3 2770, pointer 92: The used space (3144) of the child block (40454561) is not equ al to the (blocksize (4096) - free space (48) - header size (24)) /170 (of 170)bad_path: The right delimiting key [6 25399 0xba001 IND (1)] of the node (6062135) must be greater than the last (0) element's key [9 36 0xd4e2001 IND (1)] within the node. / 2 (of 89)/ 1 (of 107)block 138444801: The level of the node (0) is not corr ect, (1) expected the problem in the internal node occured (138444801), whole subtree is skipped / 3 (of 89)/ 1 (of 150)bad_path: The left delimiting key [9 36 0x89c6001 IND (1)] of the node (6062136) must be equal to the first element's key [9 36 0xd8d6 001 IND (1)] within the node. / 36 (of 150)bad_path: block 6029356, pointer 35: The used space (3264) of the c hild block (44570431) is not equal to the (blocksize (4096) - free space (892) - header size (24)) / 5 (of 89)/ 21 (of 133)bad_stat_data: The objectid (47857) is marked free, bu t used by an object [9 47857 0x0 SD (0)] bad_stat_data: The objectid (47958) is marked free, but used by an object [9 479 58 0x0 SD (0)] bad_stat_data: The objectid (48026) is marked free, but used by an object [9 480 26 0x0 SD (0)] / 26 (of 89)/105 (of 154)bad_stat_data: The objectid (61571) is marked free, bu t used by an object [47 61571 0x0 SD (0)] bad_stat_data: The objectid (61593) is marked free, but used by an object [47 61 593 0x0 SD (0)] / 75 (of 89)/ 93 (of 160)bad_stat_data: The objectid (61454) is marked free, bu t used by an object [133 61454 0x0 SD (0)] / 2 (of 5)/ 6 (of 142)/170 (of 170)bad_path: The right delimiting key [163 1 64 0x6d45001 IND (1)] of the node (62318332) must be greater than the last (0) e lement's key [163 165 0x107d0001 IND (1)] within the node. / 7 (of 142)block 90177538: The level of the node (0) is not correct, (2) expec ted the problem in the internal node occured (90177538), whole subtree is skipped / 4 (of 5)/ 27 (of 167)/ 1 (of 157)bad_path: The left delimiting key [11274 25170 0x5afc001 IND (1)] of the node (346521841) must be equal to the first elem ent's key [11274 25166 0x36d09001 IND (1)] within the node. /157 (of 157)bad_path: The right delimiting key [11274 47855 0x964e001 IND (1)] of the node (280166406) must be greater than the last (0) element's key [11275 2 5163 0x176a001 IND (1)] within the node. / 28 (of 167)block 347005902: The level of the node (0) is not correct, (2) expe cted the problem in the internal node occured (347005902), whole subtree is skipped / 5 (of 5)/ 16 (of 100)/ 10 (of 88)bad_stat_data: The objectid (25228) is sh ared by at least two files. Can be fixed with --rebuild-tree only. finished Comparing bitmaps..vpf-10640: The on-disk and the correct bitmaps differs. Bad nodes were found, Semantic pass skipped 8 found corruptions can be fixed only when running with --rebuild-tree ########### reiserfsck finished at Sun Jan 24 14:57:27 2010 ########### root@Tower:~#
January 25, 201016 yr After installing the new drive,and determining its "device" type the following dd if=/dev/sdc of=/dev/sdX bs=1M sdX must be the new drive. Be super careful here. If you give the device of an existing data drive assigned to your array, its contents will be overwritten. If you mix up the "if" (input file) and "of" (output file) you will copy the blank new drive to your old drive completely clearing its contents and any chance of getting to your data. The process of copying a 1.5TB drive will take 3 to 5 hours or more. let the process run. When it is complete, it will print the total copied and the time elapsed. Make absolutely certain you have the correct devices. Double check by their serial numbers and the use of hdparm -I /dev/sdX or smartctl -a -d ata /dev/sdX Oh yes, to copy in a telnet window you can usually highlight the text you wish to copy then a right hand click on the top menu bar will give you a "copy" option. If using "putty" highlighting text automatically copies it to the clipboard.
January 25, 201016 yr Yes, you have some corruption... as expected. Now, get some sleep, purchase a new disk. We'll continue tomorrow. I'm going to get some sleep.
January 25, 201016 yr Author Ok, I installed the drive, and started the copy process, I think. When I typed in the dd commpand, not much happened, it just went to the next line of text, no progress meter or anything like that, is that normal? It looks like below, with the underscore on the second row just blinking on and off, is it running? root@Tower:~# dd if=/dev/sdc of=/dev/sdd bs=1M _ What happens if I close/exit my telnet session? Will that screw up the copy, or will it keep going?
January 25, 201016 yr Ok, I installed the drive, and started the copy process, I think. When I typed in the dd commpand, not much happened, it just went to the next line of text, no progress meter or anything like that, is that normal? It looks like below, with the underscore on the second row just blinking on and off, is it running? root@Tower:~# dd if=/dev/sdc of=/dev/sdd bs=1M _ Yes, it is running What happens if I close/exit my telnet session? Will that screw up the copy, or will it keep going? The copy will abort. It will not keep on going. Don't close the telnet session. If you need to type something else, you can open a second telnet session. Joe L.
Archived
This topic is now archived and is closed to further replies.