March 26, 200917 yr Hi, Been running for over a year without a hitch, had a drive fail on me and replaced it as per instructions on these forums. The problem now though is that whilst it shows the new drive, the image is red it shows the new drive on top and the old drive name underneath? I can't start my array now as it tells me I have "Too many wrong and/or missing disks!". How can I get it to use this new drive and rebuild it based on what the old drive contained?
March 26, 200917 yr Hi, Been running for over a year without a hitch, had a drive fail on me and replaced it as per instructions on these forums. The problem now though is that whilst it shows the new drive, the image is red it shows the new drive on top and the old drive name underneath? I can't start my array now as it tells me I have "Too many wrong and/or missing disks!". How can I get it to use this new drive and rebuild it based on what the old drive contained? I think it has something to do with disk13, which shows "not installed" in italics, but yet has a size. Any idea what that's all about?
March 26, 200917 yr Author That's been there for nearly a year when I stuck my cache drive in slot 13, thought about it and then removed it and put it in slot 14. Didn't seem to bother anything so I left it be? Could they be related?
March 26, 200917 yr What is the story with disk13? That is why the array will not start... It thinks you have 1 missing disk AND 1 replaced disk. Once we get the answer to that question, and you explain "specifically" what you did when you replaced the drive, we can give some guidance. Did you use a different cable for the new drive? Or a different disk controller? Move it to a different slot? It is very NORMAL to see the old and new serial numbers when replacing a disk. That part is expected. I'm more curious about the disk13 "not installed" with he "red" indicator. Did you have a disk in it at one time? and then removed it? and then un-assigned it? Can you post a syslog. It will help at lot to see what the array thinks it should be seeing. Lastly, what version of unRAID are you running? Joe L.
March 26, 200917 yr What is the story with disk13? That is why the array will not start... It thinks you have 1 missing disk AND 1 replaced disk. Once we get the answer to that question, and you explain "specifically" what you did when you replaced the drive, we can give some guidance. Did you use a different cable for the new drive? Or a different disk controller? Move it to a different slot? It is very NORMAL to see the old and new serial numbers when replacing a disk. That part is expected. I'm more curious about the disk13 "not installed" with he "red" indicator. Did you have a disk in it at one time? and then removed it? and then un-assigned it? Can you post a syslog. It will help at lot to see what the array thinks it should be seeing. Lastly, what version of unRAID are you running? Joe L. Had the indicator been "red" for the past year next to disk13?
March 26, 200917 yr Author Yup, ever since I removed the drive. Too be honest I've had very little to do with the server as it's in the cupboard and just works perfectly, well upto the drive failing and trying to replace it. Anyway, the drive failed and said it was missing on screen, so I stopped the array and powered off. I then put the new drive in, powered up and .....
March 26, 200917 yr Author Can you post a syslog. It will help at lot to see what the array thinks it should be seeing. Sorry, irritating newbie how do I get a syslog? And I'm running 4.3.2 http://pastebin.com/m2f22a88f
March 26, 200917 yr Yup, ever since I removed the drive. Too be honest I've had very little to do with the server as it's in the cupboard and just works perfectly, well upto the drive failing and trying to replace it. Anyway, the drive failed and said it was missing on screen, so I stopped the array and powered off. I then put the new drive in, powered up and ..... I'm afraid of what I'm thinking. I hope you are sitting down... You have been running in a "degraded" mode for about a year. When you "removed" the disk from slot13 in the array it was EXACTLY as if it had failed. Un-assigning it is EXACTLY the same as if it was missing. It would not be removed from the array unless you pressed the "Restore" button which actually stores a new Initial configuration based on the then currently assigned and working drives. The ONLY way to remove a drive from a slot once you assign it, and not replace it with another, is to use the poorly named "restore" button to "Store a new initial configuration". Now, when drive7 failed, you had two failed drives... There is no way to rebuild from parity... at least not reliably... There is a glimmer of hope, however, depending on what was on the cache drive when you assigned it to slot13. Was it completely empty? Did you let unRAID clear it and format it in that slot before adding it to the array in that slot? If yes... you might be able to get some of the data restored on the defective drive's replacement, if not... welll... this might be a bit of a nightmare. Regardless, you might not get everything back... Basically, we need to force your server to think that disk13 was never installed. We then need to force it to think parity can be used to fix disk7 and that disk7 is the one being replaced. The problem lies in the bits that were non-zero in disk13. For every one that was non-zero, the corresponding bit in drive7 being rebuilt will be rebuilt wrong. If all you had was a few files, or if disk13 was completely empty, with just partitioning, it might just be those files that are corrupt. On the other hand, it might result in the partitioning all being corrupt when you rebuild the disk7. You will need to follow very specific steps to have any hope of recovering anything from the parity data... Don't press any buttons until instructed. First, fill me in on what you remember of the disk in slot13 and how it was added. (I know it was a year ago... but try to remember) Lastly, how did the disk7 fail? Are we certain it is really failed? (It would be really great if it could be read...) The reliability of a RAID array is completely dependent upon a failed disk being replaced before a subsequent failure of a second disk. Unbeknown to you, you have had a "failed" disk in your array this past year. Now, you have a second failed drive. The only hope we have is that most of the disk13 was probably zeros. Fill in some more details and we'll advise further. Joe L.
March 26, 200917 yr Author Disk13 was a 500GB disk added to the array in standard fashion, and then I copied across about 40GB of music to it. I then decided that that was a waste, and used it for the cache drive. Actually that's a crock, it was a 120GB drive that had 1GB of music on it. I then took it out and used it elsewhere. My disk7 just never gets added to the array, it always comes back saying missing. It can't be mounted because of bad blocks or some such. I guess I could still get the data off it with some work. Should I add disk7 back into the machine ?
March 26, 200917 yr Author I've stuck the new drive in slot 8 (but done nothing with it yet) and put disk7 back in and now it doesn't say it's missing, but it's unformatted? And you'll notice (and I never did :-) ) that disk13 now gives you some details about what it was When I hit stop, then I get missing
March 26, 200917 yr It is looking unlikely that you are going to magically get the original disk7 working by ordinary means. Do you have a backup or alternate means of recreating (e.g., could you rerip a bunch of movies and/or CDs?) If the answer is ... "there is no way to recover and there is no backup", I have to tell you the prognosis is not good for FULL data recovery, but that the prognosis is good for a partial recovery. In experimenting with getting disk7 to be recognized in the array, BE VERY CAREFUL THAT YOU DO NOT ADD OR DELETE ANY DATA TO THE ARRAY. Every piece of data you add or update can make recovery less accurate. Before proceeding, a short "I told you so." This will be the only one and then I will do my best to help you recover. I have been a strong advocate for everyone doing a monthly parity check. I am near certain that had you attempted to run a parity check you would have been told that you can't with a missing disk. The button would likely have been grayed out. Periodic parity checks are your best way to ensure that your array is recoverable in the event of a disk failure. And now, on to your normally scheduled post ... Please know that there are some techniques available to keep a limping drive running for a few minutes (or hours or even days) in order to recover data from a failing drive. The most common one is to put the offending drive in the freezer. Do some searching and try these techniques. If you can get the drive to recognize, make sure you already have prepared room ON ANOTHER MACHINE and immediately begin copying your most valuable data over the net (or on a separately mounted disk on the unRAID server, but this starts to get complicated). Using this technique does not in any way stop you from later doing to recovery described below. This is your only chance of a full data recovery. I think it is important to understand your current state. First a brief parity tutorial ... On a "clean array", parity is set to 1 or 0 on a bit by bit basis in order to make the sum or all corresponding bits on all disks EVEN. (It might actually be done on a larger than bit by bit basis, but conceptually this is what is happening) disk1: 1 1 1 1 disk2: 1 1 0 1 disk3: 0 0 1 1 disk4: 1 0 0 0 parity: 1 0 0 1 If this isn't clear, read this post. Now if disk4 were to fail or be pulled from the array for any reason (equivalent to what happened with your disk13), unRAID would simulate that disk by reading parity and all of the other disks and recomputing disk4 in real time. (This mode is sometimes called "degraded" mode because accesses to disk4 would be slower than normal. But if you never look at disk4, there is no degradation. I like to just call it simulated mode). disk1: 1 1 1 1 disk2: 1 1 0 1 disk3: 0 0 1 1 disk4: 1 0 0 0 parity: 1 0 0 1 I show it in blue because disk4 isn't there. It is being simulated. Now if disk2 were to fail (equivalent to your disk7 failing), you no longer have enough information to simulate disk disk4 or disk2. disk1: 1 1 1 1 disk2: ? ? ? ? disk3: 0 0 1 1 disk4: ? ? ? ? parity: 1 0 0 1 With 2 disks missing, you have no way to figure out what disk2 or disk4 values should be. Recovery is not possible. The situation improves a bit if you know that disk4 was added to the array and never formatted. In that case, disk4 would be completely full of binary zeros. In that case: disk1: 1 1 1 1 disk2: ? ? ? ? disk3: 0 0 1 1 disk4: 0 0 0 0 parity: 1 0 0 1 unRAID would then have enough information to reconstruct disk2. The tricky part is telling unRAID that disk4 is full of zeros when there is really no disk in the array that looks like that. But there is a way to make unRAID believe that disk4 is completely out of the array (equivalent to being all zeros) while maintaining parity. But notice that in my example, disk4 is NOT all zeros. The first bit is a one, and all the rest are zeros. So reconstruction would not be perfect disk1: 1 1 1 1 disk2: 0 1 0 1 disk3: 0 0 1 1 parity: 1 0 0 1 The first bit is wrong, but the rest are correct. Your disk13 was formatted, and had about 1G of data written to it. That means that when you rebuild disk7 (after tossing disk13 from the array without losing parity), the corresponding parts (about the first 1G) on your restored disk7 are going to be corrupt. But everything above the 1G mark should be good. Make sense? Part of the corrupted portion will be what I'll call the "housekeeping" part of the drive, where it keeps the formatting information. For that reason, after reconstruction, your disk will likely appear unformatted. There is a tool called "reiserfsck" that is designed to fix broken formatting problems, and will likely be able to rebuild that part of the disk and allow you to recover a large percentage of your data.
March 26, 200917 yr Author Before proceeding, a short "I told you so." This will be the only one and then I will do my best to help you recover. Fair play, I know it's my fault and luckly if it can't be saved the only real data on this disk is TV shows that I can rip again from DVD or replace in other ways ;-) And as for monthly parity checks, well guess who'll be doing them from now on I'm at work now it being 09:15 Friday morning here in Sydney so I can't attempt anything on the array until tonight, but please fire away any suggestions and many thanks for all your help. (edit) Actually, could I buy a new 1TB drive and stick that in disk13 along with the new one I got the other day in disk7 and just let it rebuild them both? Would that work or it that too simplistic?
March 27, 200917 yr I felt that this problem was important enough to warrant its own FAQ entry, "If a drive fails, will I have full access to my unRAID array?" I've generalized it somewhat, and put it near the top of the FAQ. I also included a link back to here, because your disaster will make an excellent example for others, but I do understand that that may make you uncomfortable, with a bit of a spotlight on your problem here, so I'll remove it if you wish. Your story does make a good warning for others though. I suspect there may be a few other users with a similar situation, who have been downplaying or procrastinating about a single red ball, thinking that, for whatever reason, it's not that important. Actually, could I buy a new 1TB drive and stick that in disk13 along with the new one I got the other day in disk7 and just let it rebuild them both? No, sorry. The whole idea as Brian illustrated, is that you can lose any one disk, and use all of the others to rebuild it. But that means you can never rebuild more than one disk at a time. The only (rather iffy) idea I have, over the excellent advice you have been given, is if Joe can possibly come up with a way to completely 'pre_clear' Disk 13, in a way that also updates the parity info. If he can think of a way to clear an array disk that updates parity, but does not cause too many Reiser file system errors (manually unmount md13 first?), then you might be able to improve the accuracy of the Disk 7 rebuild.
March 27, 200917 yr Author In terms of embarrassing things that have been associated with my name over the last 43 years, this is less than nothing. My actual disk7 has died totally now in a flurry of grinding noises so I'm hoping that we can recover something, but worst case is I have to dig the DVDs back out of the garage and redo them and that's not the end of the world. Once again though, many, many thanks to all who've taken the time to help me out here, and hopefully it will all work out in the end!
March 27, 200917 yr The only (rather iffy) idea I have, over the excellent advice you have been given, is if Joe can possibly come up with a way to completely 'pre_clear' Disk 13, in a way that also updates the parity info. If he can think of a way to clear an array disk that updates parity, but does not cause too many Reiser file system errors (manually unmount md13 first?), then you might be able to improve the accuracy of the Disk 7 rebuild. An exceedingly good idea. I like the way you think! I don't think that Joe L. would have to update the preclear script. Just one of those ... dd if=/dev/zero of=/dev/disk13 Of course disk7 would have to be alive (or unRAID would have to think its alive), which sounds like not possible. I think that, sadly, this will go down as a complete loss of disk7. :'( :'( :'( Moral of the story - don't ignore the red balls on the management console. If there is one, even if the array seems fine and dandy, unRAID is already using its redundancy to simulate the disk, and one more failure and bam!, your data on both disks is gone. A tangential question came to mind as I was writing a response in this thread ... What would happen, if the monthly cron job to check parity ran, and the array that was simulating a disk? Would it still go through the motions, reading simulated sectors (impossible to get errors, right?). Or would it get some error and fail immediately? Would the user get any indication at all that the parity check failed or didn't occur, like sync errors = 999999? Would the last parity check date get updated? Anyone know or have a theory?
March 28, 200917 yr Author So, before I give up and hit Restore, which I guess is where we're headed right?, is there anything I can try or should I just slap myself on the wrist, look shame faced in the mirror, resign myself to re-ripping the DVDs and just hit the button?
March 28, 200917 yr So, before I give up and hit Restore, which I guess is where we're headed right?, is there anything I can try or should I just slap myself on the wrist, look shame faced in the mirror, resign myself to re-ripping the DVDs and just hit the button? If you go back and re-read my post (not the last one, but the one before that), you'll see that there is a way that will likely recovery most of your data. I will post back a little later today with directions.
March 28, 200917 yr Here are the series of steps I think give you the best chance of data recovery. You willl need another 1T drive. *** DO NOT RUN OR ATTEMPT TO RUN A PARITY CHECK IN ADVANCE *** 1. Power down and remove the damaged disk and install the new replaceement disk 2. Power up the server, the array will not start (if it does, stop it immediatelly) 3. Go to the devices page, unassign disk13, and assign your replacement disk to disk7 4. Return to the main page 5. Press the restore button (usually a bad idea, but necessary in your case) Do not start the array. 6. Open a telnet session, and enter the command ... mdcmd set invalidslot 7 (this tells unRAID that when the array is started, to rebuild disk7 instead of rebuilding parity, which is the default) 7. Go back to the unRAID Web GUI / main tab, and start the array. 8. You should see the write count on disk7 growing, and the read counts on the other disks growing (if you refresh the page a few times) 9. Let it complete. 10. Likely in the end the disk7 will still show unformatted (due to the corruption caused by disk13) - or it may be gobbty gook. DO NOT FORMAT THE DISK! 11. You now need to run reiserfsck. See this wiki for instructions. After running it I'd suggest a reboot before looking at the recovered contents. 12. Look at disk7. Realize it won't be perfect. Your root directory might be a mess - but explore every subdirectory looking for signs of valid data. Good luck. Check out this thread of a user that needed to do something similar. I gave him a similar set of instructions to what I posted above (see here). Tibbar had some interesting problems running reiserfsck but with Joe L.'s help got it to work. He had a successful outcome. You might also look at this thread about this incredibly careless newbie that put his data disk into the parity slot and started the array. None of us are perfect. Good luck!
March 28, 200917 yr Author Thank you so much, it's running now and I'll report back in approx 324.5 minutes! Update So I officially have a new hero! Followed your advice detailed above all good until I got to point 11 and it failed immediately with bad block 0. So I had to recreate the superblock and then reiserfsck --rebuild-tree which took another 7 hours but now I have my drive back although it's in a hell of a state, as far as I can see it's at least 95% there! I can't thank you enough for helping me recover this data. I'll run through it in full tomorrow after work and then report back with the final findings on what was recovered and lost in percentage terms.
March 29, 200917 yr Author So what actually happened was disk7 came back with a single directory called lost+found. In this directory were 179 subdirectories with names like 165_6731 or 56_988 and 47 files in the root totalling 478GB. Going through these directories I discovered that everything within them was fine and I just had to work out what to rename the directory, i.e. 165_9987 was TV show "Ashes to Ashes" and 65_988 was "Dexter" etc. Now I had all my directories back and 4.36GB of unknown files in the root which I suspect must be a ripped DVD but the individual file sizes are way to small. Anyway to only lose 4.36GB from nearly 480 is awesome and from now on I promise I'll be a good unRAID citizen and do my housekeeping on a regular basis
March 30, 200917 yr So what actually happened was disk7 came back with a single directory called lost+found. In this directory were 179 subdirectories with names like 165_6731 or 56_988 and 47 files in the root totalling 478GB. Going through these directories I discovered that everything within them was fine and I just had to work out what to rename the directory, i.e. 165_9987 was TV show "Ashes to Ashes" and 65_988 was "Dexter" etc. Now I had all my directories back and 4.36GB of unknown files in the root which I suspect must be a ripped DVD but the individual file sizes are way to small. Anyway to only lose 4.36GB from nearly 480 is awesome and from now on I promise I'll be a good unRAID citizen and do my housekeeping on a regular basis Glad to here you got virtually all of your data back. This case proved to be one of the most "interesting" I have read. I have gone through a similar experience and had to recover my data. I ended up getting almost all of it back but had to rename some stuff. The RFS has proven to be robust in recovering data and very good in my experience.
March 30, 200917 yr So what actually happened was disk7 came back with a single directory called lost+found. In this directory were 179 subdirectories with names like 165_6731 or 56_988 and 47 files in the root totalling 478GB. Going through these directories I discovered that everything within them was fine and I just had to work out what to rename the directory, i.e. 165_9987 was TV show "Ashes to Ashes" and 65_988 was "Dexter" etc. Now I had all my directories back and 4.36GB of unknown files in the root which I suspect must be a ripped DVD but the individual file sizes are way to small. Anyway to only lose 4.36GB from nearly 480 is awesome and from now on I promise I'll be a good unRAID citizen and do my housekeeping on a regular basis ;D Enjoy your array!
.thumb.jpeg.49f968af52d6b35f4766eff2a95fd42e.jpeg)
Archived
This topic is now archived and is closed to further replies.