July 11, 200916 yr Oh yes, the reason I suggested doing this to the new drive, rather than ovewriting the old, is so you do nothing more to modify its contents at this time. In effect, it is put aside during this attempt to get the reconstructed data back onto a new replacement drive. Joe L.
July 13, 200916 yr Author Oh yes, the reason I suggested doing this to the new drive, rather than ovewriting the old, is so you do nothing more to modify its contents at this time. In effect, it is put aside during this attempt to get the reconstructed data back onto a new replacement drive. Joe L. Interesting approach Joe. Unfortunately, the disk to be added was 250GB, whereas the so-called unformatted one was 1TB. The 1TB drive contained relatively few files. Probably some torrents on their way in, other than that I have not noticed anything missing so far. What happens if I do a "trust my parity drive" with the smaller one installed? The 1TB drive was _certainly_ not more than 25% full. Think I can still do this trick?
July 13, 200916 yr Oh yes, the reason I suggested doing this to the new drive, rather than ovewriting the old, is so you do nothing more to modify its contents at this time. In effect, it is put aside during this attempt to get the reconstructed data back onto a new replacement drive. Joe L. Interesting approach Joe. Unfortunately, the disk to be added was 250GB, whereas the so-called unformatted one was 1TB. The 1TB drive contained relatively few files. Probably some torrents on their way in, other than that I have not noticed anything missing so far. What happens if I do a "trust my parity drive" with the smaller one installed? The 1TB drive was _certainly_ not more than 25% full. Think I can still do this trick? It does not work that way. Do NOT try what I suggested. The new disk must be as large or larger then the original. sorry. So again... do NOT swap the assignment of the drives. Joe L.
July 13, 200916 yr I am wondering if this could be HPA related? Radiopaque, what motherboard are you using? If the hotswap operation trashed the beginning of the disk - the motherboard might have thought this would be a great time to add HPA. And then you could have all sorts of problems with reiserfsck. Regardless, given that fact that there was little data on the disk, it might make sense to just call it a loss, reformat it, and rebuld your parity. Anyone reading this - UNRAID DOES NOT SUPPORT HOT-SWAP, EVEN IF THE HARDWARE DOES. Save yourself a lot of time and aggravation and avoid the impulse to try it!
July 13, 200916 yr I am wondering if this could be HPA related? Radiopaque, what motherboard are you using? Good idea, Brian, but it is not a Gigabyte board - he has "A M I OEM..." in early ACPI table, not "GBT GBTU...". It's a good idea because that is one known agent that modifies (without asking) the partition table, but the Gigabyte HPA 'cylinder grab' only subtracts a few LBA's from the ending LBA of the last partition, about 2 megabytes worth. Apart from adjusting that one number in the partition table, I don't see any other writes to the early sectors. It also adjusts the firmware's report of the MAX_SIZE of the drive, which can confuse a file system, but I don't see how that would cause overwrites of the first sectors either. Radiopaque: just wanted to make sure you saw my note to you, here. His thread very much parallels yours, except he is benefiting from your misfortunes, actually we all are. We were not aware of the danger in this 'minefield'. You may want to follow his adventures too, as he is probably following yours. To summarize, so we are on the same track, and please correct me where I'm wrong: * You ran the "reiserfsck --rebuild-sb /dev/md6" command, and exited when you encountered the "Do you want to rebuild the journal header? (y/n)[n]:" question. * You then re-ran the command with the added parameter "--no-journal-available", which was unfortunately wrong, but none of us knew that at the time. It may have caused some additional corruption. The command appeared to finish correctly ... * But when you ran the reiserfsck check, found that the journal params were wrong. * You then re-ran the rebuild of the superblock, apparently correctly. * You then started another check, and since no errors were immediately reported, it must have been satisfied with the superblock and journal. It did find an error in a bitmap, which means a little corruption has been found and you would have been told on completion to rerun with either --fix-fixable or --rebuild-tree. But then it aborted with the fatal "Bad root block 0. (--rebuild-tree did not complete)", particularly odd because you were only running a check, not rebuild-tree. * You then ran "reiserfsck --rebuild-tree /dev/md6", but no leaves were found (trees need leaves!), so very bad news. * You then ran "reiserfsck --scan-whole-partition --rebuild-tree /dev/md6", good idea but no better results, which is worse news. I found a number of cases of the "Bad root block 0", but many were dead ends. That is, after some initial advice, there was no further help and no report of success, so likely resulted in a loss of data. There were a few cases, where a repeat of the command would (inconsistently) work. That's worth one more attempt. But in general, as the doctors would say, this message and/or a rebuild-tree that did not finish is often associated with negative outcomes. If you would like to try recovering files, there are several choices (no guarantees). Test Disk (free, run from a live CD such as the System Rescue CD) can often recover partition or data. Stellar Data Recovery and UFS Explorer have downloadable tools that have been able to recover some of the lost files on a fatally damaged Reiser file system. Neither are free. There are probably others too. The reiserfsck advice at the end is probably not useful, depending on whether there has been a significant partition change. Try running "fdisk -l /dev/sdb" to compare its partition table with this one on sda, since both drives are very similar. If identical, then there is no point in looking for a superblock elsewhere. And NameSys is essentially gone, the people have pretty well scattered, or are in jail.
July 15, 200916 yr Author Hi, Many thanks for all the time you guys have put into this already. I feel that I'm almost out of options, with testdisk being one of the few that remain. Assuming that does not give any joy: what are the odds of getting some (or even most) data back by re-calculating the disk from parity? I would be doing this on the _same_ drive! Thanks, Mark
July 16, 200916 yr what are the odds of getting some (or even most) data back by re-calculating the disk from parity? I would be doing this on the _same_ drive! The problem is, how do you know if the damage is 'recorded' in the parity info or not. There is a good chance that when the damage occurred, it looked like an ordinary write to unRAID, which then updated the parity info accordingly. Yes, rebuilding the drive is an option, but I would only try a rebuild on to a _different_ drive, not the same one. Or image the drive first, so you can restore it back, or attempt data recovery on that image of the drive. However, you are right if you are thinking that rebuilding the drive *might* recover an earlier version of the drive, maybe still not perfect, but *possibly* in better shape than its current state.
July 16, 200916 yr Sounds like you have exhausted virtually all options to recover data from that disk. If so, I can see no harm in reconstructing from parity directly back to that disk. Maybe it will help you recover and maybe not. Obviously there would be no going back.
July 16, 200916 yr what are the odds of getting some (or even most) data back by re-calculating the disk from parity? I would be doing this on the _same_ drive! The problem is, how do you know if the damage is 'recorded' in the parity info or not. There is a good chance that when the damage occurred, it looked like an ordinary write to unRAID, which then updated the parity info accordingly. Yes, rebuilding the drive is an option, but I would only try a rebuild on to a _different_ drive, not the same one. Or image the drive first, so you can restore it back, or attempt data recovery on that image of the drive. However, you are right if you are thinking that rebuilding the drive *might* recover an earlier version of the drive, maybe still not perfect, but *possibly* in better shape than its current state. Since you have been running reiserfsck on /dev/md6, all the changes you've made attempting to repair it are already reflected on the parity disk. Resonstructing a new disk from parity will give exactly the same contents as you currently have. Joe L.
July 16, 200916 yr Bummer. In hindsight, it might have been smart to have started this whole adventure with a read only parity check.
July 16, 200916 yr Here's a possible plan of attack. 1. Make an image of the existing disk6. If it is at all possible to recover data, it will be used. Make a debugreiserfs image of the structure of the damaged drive. It might be possible to use it to figure out what happened. (I've personally never used this tool, but am willing to learn, and I have space to unzip its output onto a spare drive.) A debug image is a dump of the directory structure, but not the bulk of the files. Note, reiserfs does store small parts of the ends of files in spare space in the directory structure, so small parts of your files will be in the output if you do this) the command is: debugreiserfs -p /dev/md6 |gzip -c > /mnt/disk1/md6_debug.gz Now, you probably do not have gzip installed on your server, but you can download it from here: http://slackware.cs.utah.edu/pub/slackware/slackware-12.2/slackware/a/infozip-5.52-i486-2.tgz and install it with installpkg infozip-5.52-i486-2.tgz Then you can run the debugreiserfs command. If you do not have space on disk1 for the output, choose one on your server that does. You can share the file-system debug structure for analysis if there is nothing sensitive on it. debugreiserfs seems to have the ability to extract blocks from a specific file... This thread shows its use (I have no experience at all in doing this) http://www.mail-archive.com/[email protected]/msg13010.html You can possibly find and reconstruct something critical this way. Joe L.
July 16, 200916 yr If you have completely given up hope, there is one more thing to try. You can re-format the drive. It should completely rebuild the superblock and the internal bitmaps. The bitmaps will be initially empty. Then, you can run "reiserfsck --rebuild-tree --scan-whole-partition" to scan the entire disk and let it recover what it can. The commands would be: mkreiserfs /dev/md6 reiserfsck --rebuild-tree --scan-whole-partition /dev/md6 As an experiment, I just tried this on a small disk here. It had a reiserfs file system on it, and several files at the root level, and several in a "Movies" folder. The results were as follows: all files were found and rebuilt and put in a lost+found folder. They did not have their original names, but instead were named after their block ID in the FS. I'd need to rename them, but I'd have my movies. So... even after a re-format you can still use the rebuild-tree scan-whole-partition to recover the files. (as long as you do not overwrite them with new contents between the reformat and the rebuild step) Joe L.
July 19, 200916 yr Author Great idea, Joe! I concur. It is not likely that this disk contains a huge amount of data, and if any, if should be pictures (they've got those weird IMG_xyz.DNG names anyway) or large music files (flac) or movies. Also, the disk was at the end of the array, which was only moderately full. I am hoping the disk was not very full when disaster struck. Mark
July 20, 200916 yr Author Hi all, It's all done. Nothing in the lost+found directory! Now, I would say that at least some junk would appear there, but no... I will check the output of the chkdsk command when I get home, but it somehow finished and after a mount of md6, all I have is an empty lost+found. With a standard high-water allocation, is there a way to backtrack whether this is at all to be expected? Thanks, Mark
July 20, 200916 yr Hi all, It's all done. Nothing in the lost+found directory! Now, I would say that at least some junk would appear there, but no... I will check the output of the chkdsk command when I get home, but it somehow finished and after a mount of md6, all I have is an empty lost+found. With a standard high-water allocation, is there a way to backtrack whether this is at all to be expected? Thanks, Mark Did you use the "--scan-whole-partition" option? Joe L.
July 21, 200916 yr Author It's all done. Nothing in the lost+found directory! Did you use the "--scan-whole-partition" option? Yup. And after so much help, I am sorry to say that I think this is the end of whatever data was on that drive. I'm hoping that it is not much, I haven't seen anything missing so far, aside from a torrent or two that was not even complete. The drive has now been formatted by unraid, and the parity drive has been recalculated. No way back. I'd like to thank Rob and Joe, and everyone else involved in this rescue attempt, for the many ideas and explanations. Best regards, Mark
July 21, 200916 yr It's all done. Nothing in the lost+found directory! Did you use the "--scan-whole-partition" option? Yup. And after so much help, I am sorry to say that I think this is the end of whatever data was on that drive. I'm hoping that it is not much, I haven't seen anything missing so far, aside from a torrent or two that was not even complete. The drive has now been formatted by unraid, and the parity drive has been recalculated. No way back. I'd like to thank Rob and Joe, and everyone else involved in this rescue attempt, for the many ideas and explanations. Best regards, Mark Maybe just to add my experience: I had a drive after a servermigration as unassigned, but showing reiserfs, unmountable, missing superblock etc. I also did all those recoverythings with the same result: A Lost&found was created, that was empty. I am not 100% sure, but it MIGHT be possible, that this was my former parity drive - that I replaced by a bigger one some time ago - while I "tought" I had reasigned it as data-drive... If anyone would like to test: unassign a paritydrive - it shows reiserfs (why? thought it's only binary?) and is empty&unnmountable giving superblock failure. reiserfschk gives same result as above. So I still don't know, if it was a fully loaded datadrive or my old paritydrive - but I have formatted it too and reused. Maybe this helps someone else and maybe there is a way to show "paritydrives" not as "reiserfs-drives" to avoid confusion? At least unformatted would be preferred -although it's obvious, that such a drive cannot be mounted properly; but for a user, a drive showing a filesystem means there supposed to be files on it.. What I learned: Be very careful with the paritydrive and take snapshots of the driveassignments everytime you change something on your systems... So right now while migrating to another unraid server, I got a red drive in the array - I am in the process of moving from windowsraids to unraid, reusing the drives. Now I am fearing about replacing this drive and getting errors again... thinking about buying a new one to be safe ... so a question from my side comes up: What is the best way to monitor unraids "health" state - especially for the drives? Is there an automatic possibility instead of checking the systems AFTER an error accured? (Assuming the servers are already running, so preclear is not an option plus it is a onetime activity and does not help if drives become bad after a certain time of operation)?
July 21, 200916 yr It's all done. Nothing in the lost+found directory! Did you use the "--scan-whole-partition" option? Yup. And after so much help, I am sorry to say that I think this is the end of whatever data was on that drive. I'm hoping that it is not much, I haven't seen anything missing so far, aside from a torrent or two that was not even complete. The drive has now been formatted by unraid, and the parity drive has been recalculated. No way back. I'd like to thank Rob and Joe, and everyone else involved in this rescue attempt, for the many ideas and explanations. Best regards, Mark Maybe just to add my experience: I had a drive after a servermigration as unassigned, but showing reiserfs, unmountable, missing superblock etc. I also did all those recoverythings with the same result: A Lost&found was created, that was empty. I am not 100% sure, but it MIGHT be possible, that this was my former parity drive - that I replaced by a bigger one some time ago - while I "tought" I had reasigned it as data-drive... If anyone would like to test: unassign a paritydrive - it shows reiserfs (why? thought it's only binary?) and is empty&unnmountable giving superblock failure. reiserfschk gives same result as above. So I still don't know, if it was a fully loaded datadrive or my old paritydrive - but I have formatted it too and reused. Maybe this helps someone else and maybe there is a way to show "paritydrives" not as "reiserfs-drives" to avoid confusion? At least unformatted would be preferred -although it's obvious, that such a drive cannot be mounted properly; but for a user, a drive showing a filesystem means there supposed to be files on it.. What I learned: Be very careful with the paritydrive and take snapshots of the driveassignments everytime you change something on your systems... So right now while migrating to another unraid server, I got a red drive in the array - I am in the process of moving from windowsraids to unraid, reusing the drives. Now I am fearing about replacing this drive and getting errors again... thinking about buying a new one to be safe ... so a question from my side comes up: What is the best way to monitor unraids "health" state - especially for the drives? Is there an automatic possibility instead of checking the systems AFTER an error accured? (Assuming the servers are already running, so preclear is not an option plus it is a onetime activity and does not help if drives become bad after a certain time of operation)? You bring up some very good points. It is fairly easy to understand why a "parity" disk might look to an OS as if it had a reiserfs on it. Let's take the simplest case, a single data drive, and a parity drive first: unRAID uses even parity. Therefore, for every bit set to a "1" on the data drive, the parity drive must also set the same bit position to a "1" in order to have an even number of "1"s. For each bit position set to a "0" on the data drive, the parity drive must also have a "0" to have an even number of "1"s. If you think about that for a second or two you will see that the parity drive in this single-data-drive situation will be an exactly identical copy of the single data drive. Everything will be identical, the formatting, and the data. This is identical to having a completely mirrored drive... It will look like a reiserfs, because it is. When you add a second data drive, the reiserfs superblock area on the parity drive will have all the bits flipped, and probably not recognized as a file-system, but if you add one more drive, (or have any "odd" number of data drives) you will again have the initial bits on the "parity" drive in the reiserfs superblock area looking a lot like a valid reiserfs superblock for a reiserfs file-system. The only way to be certain of which drive is the parity drive is to keep good records, and to take screen shots, or to save system logs. (the drive assignments are logged) You should (must) take a new screen shot, or make a new record every time you re-configure your drives. File the screen-shot away, fold it up and put it in the server so you do not lose it, put it in an envelope taped to the cover... It might just save you from hair-loss if you find yourself in a recovery situation. Joe L.
July 22, 200916 yr Author I used to have the naive idea that unraid would take care of the drive layout, as it has the UIDs recorded. To a certain extent it does, but I am surprised that we still need to make screenshots after a change in drives. I had really hoped that the UIDs are enough to provide unraid with an idea which drive is located where, and to enable unraid to recognize a new drive even if it is inserted where a known drive used to be (e.g., /dev/sde1)! But even with these quirks (that possibly make more sense to people who have a better understanding of the system than me) I still like unraid better than RAID
July 22, 200916 yr I used to have the naive idea that unraid would take care of the drive layout, as it has the UIDs recorded. To a certain extent it does, but I am surprised that we still need to make screenshots after a change in drives. I had really hoped that the UIDs are enough to provide unraid with an idea which drive is located where, and to enable unraid to recognize a new drive even if it is inserted where a known drive used to be (e.g., /dev/sde1)! But even with these quirks (that possibly make more sense to people who have a better understanding of the system than me) I still like unraid better than RAID unRAID does recognize when disks have been changed and when a new disk replaces one it had when last booted. It will say something like "upgrading disk" if a disk has been changed. It will not start until you check the checkbox under "Start" and then start the array by clicking "Start" It can also deal with you moving data disks around in the array. It will again not start, but present you a warning that it will just record the new positions of the disks when you start it. (in this case, I think it shows the old and new model/serial numbers of the drives in a given slot) It does this only if you move disks using the same set of ports on the disk controllers as previously used. (as long as you don't move the parity drive I think it will allow you to start the array after presenting this warning) It does not automatically deal with disks showing up on disk hardware it never had before, or never before used disk-controller ports. (You add a new disk controller card and move a disk to it) When you do that, it simply leaves the disks on the new hardware unassigned waiting for you to go to the devices page to assign them. There are three main reasons for a screen shot of your disk assignments: The first reason for the screen shot is so you can put the disk that used to be in a specific logical "slot" in the array back on the same slot if needed. This is important because the disk allocation rules and shares are defined in terms of "disk1, disk2, disk3, etc." The second reason for the screen shot is if you find your flash drive has failed. If that is the case, the superblock held on it in config/super.dat is lost. It is in that file where unRAID keeps track of the model/serial numbers of your assigned drives and the configuration of your drives. If you have a screen-shot you can quickly get the array running with a replacement flash drive. A third reason for the screen shot is to be able to deal with new releases on Linux more easily. There have been a few times when a new release of Linux scans the hardware on your MB in a different order, and the disks are assigned different IDs on the bus. When you upgrade the unRAID version, and if it contains a newer release of Linux, and if it assigns the controller hardware new/different IDs, you can re-assign the drives as needed to their original slots on the array. Joe L. P.S. You are not alone in liking unRAID better than standard RAID for home media storage.
July 22, 200916 yr You bring up some very good points. It is fairly easy to understand why a "parity" disk might look to an OS as if it had a reiserfs on it. Let's take the simplest case, a single data drive, and a parity drive first: unRAID uses even parity. Therefore, for every bit set to a "1" on the data drive, the parity drive must also set the same bit position to a "1" in order to have an even number of "1"s. For each bit position set to a "0" on the data drive, the parity drive must also have a "0" to have an even number of "1"s. If you think about that for a second or two you will see that the parity drive in this single-data-drive situation will be an exactly identical copy of the single data drive. Everything will be identical, the formatting, and the data. This is identical to having a completely mirrored drive... It will look like a reiserfs, because it is. When you add a second data drive, the reiserfs superblock area on the parity drive will have all the bits flipped, and probably not recognized as a file-system, but if you add one more drive, (or have any "odd" number of data drives) you will again have the initial bits on the "parity" drive in the reiserfs superblock area looking a lot like a valid reiserfs superblock for a reiserfs file-system. The only way to be certain of which drive is the parity drive is to keep good records, and to take screen shots, or to save system logs. (the drive assignments are logged) You should (must) take a new screen shot, or make a new record every time you re-configure your drives. File the screen-shot away, fold it up and put it in the server so you do not lose it, put it in an envelope taped to the cover... It might just save you from hair-loss if you find yourself in a recovery situation. Joe L. Joe, yes, I quite understand what you say and it sounds logically ;-) - to be honest it's way too long ago when I was dealing with basic structures on drives - but I think not every byte is really owerwritten even on the parity disk. How about e.g. the drive signature - this was something like AA55 in DOS-times to show a valid drive is present...? - Something like that is already used to verify if the drive is cleared properly - can't a different signature be used to designate it as a paritydrive ? Could then be handled by the app or shown, e.g. if a user tries to use it in the GUI? Similar to your preclear script, that does not allow to clear an assigned drive and can check the precleared status... Nothing that is required for basic function, but would give more safety to the user ... warnings in the manual or the forum is one thing and protectioncode in an app the other ;-) btw, I had several copies of my unraid config in the past and tried to find the last position where the drive was used - I found a record where it was a parity drive, but no record as datadrive afterwards - was sure, that I had no "dead disks" outside the array - but "maybe" I had? ;-) PS: Me too I like Unraid - that's why I migrate my 24 windows Raid-5 drives ;-) (and really awaiting tom's fix for the 17+ disks bug...)
July 22, 200916 yr How about e.g. the drive signature - this was something like AA55 in DOS-times to show a valid drive is present...? - Something like that is already used to verify if the drive is cleared properly - can't a different signature be used to designate it as a paritydrive ? Could then be handled by the app or shown, e.g. if a user tries to use it in the GUI? Similar to your preclear script, that does not allow to clear an assigned drive and can check the precleared status... I like this idea, could improve the safety. It would help unRAID to precisely identify which is the parity drive, if superblock and other config info is lost, as well as keep the user from assigning the wrong drive to the wrong slot. There certainly is room for signature info in the earliest tracks, since ReiserFS leaves quite a few empty sectors at the beginning. Or another idea, why not assign a special partition table identifier for the unRAID parity partition, such as 88. Right now, we use 83, which is for native Linux file systems such as Reiser and ext3, and 82 is used for Linux swap partitions. As far as I can tell, 88 is available.
Archived
This topic is now archived and is closed to further replies.