July 25, 200916 yr Hey all, I've been using UnRaid problem free for some time now, but now it's happened and I've got my first failed drive. However, I think I may have messed things up trying the recover things. What happened: 1. One of my hard drives had a red light indicating a problem from the unRaid interface. I shut down the raid array to prevent risk of other hard drive failure. At this point, the hard drive serial of the problem drive was still showing up correctly on the unRaid web interface. 2. I boot up the computer again, this time I notice the instead of a red ball with the drive serial showing up, it just says drive missing. 3. I power down, and I insert a new hard drive in place of the old one as mentioned in all the documentation (same SATA) cable. 4. UnRaid rightfully complains that I accidentally inserted a hard drive greater than the parity hard drive (parity was 750GB, I inserted a 1TB drive). I shut down the drive. 5. I go out and buy a new hard drive (same size as the failed drive 500GB) and insert it inside and boot up. 6. I notice that the UnRaid server has problems getting to the regular interface. I noticed that it sometimes says 'mounting' and is freezing. I try several times to no avail. Now if I insert the new drive, the computer gets the root login, but the keyboard is frozen, and you cannot access the web interface. 7. I remove the replacement drive, and it boots up fine. However, it now complains that the replacement drive is missing (which I'm worried about, since it might imply that my original drive is not longer there) 8. I may have clicked on the 'restore' option, at one point, to see if that helped, but I do not believe it did. However, I have not brought the array online since. Normally according to documentation an option to start up and 'rebuild' the array should appear, but I never got it. I wonder if inserting the larger drive size caused an untested case to occur. 9. As it stands now, I have the option of just starting up the system again, but it seems like if I do that, i'll detect all the hard drives as new, and reset the existing parity. I have not started up the array where parity sync runs yet, so I think all the data is intact. (not sure here) Ideally, I wish I could just 1. Get to the point where the system has 1 failed drive, so the system is unprotected, but it runs of parity information so I can access the data. 2. Get to the point where inserting a new drive to replace the old drive doesn't cause the system to freeze. As luck has it, I lost my most important drive with all my old files. Any help would be greatly appreciated. Thanks. -Daniel
July 25, 200916 yr Do not bring the array online!!! Wait for help. Hey all, I've been using UnRaid problem free for some time now, but now it's happened and I've got my first failed drive. However, I think I may have messed things up trying the recover things. What happened: 1. One of my hard drives had a red light indicating a problem from the unRaid interface. I shut down the raid array to prevent risk of other hard drive failure. At this point, the hard drive serial of the problem drive was still showing up correctly on the unRaid web interface. 2. I boot up the computer again, this time I notice the instead of a red ball with the drive serial showing up, it just says drive missing. 3. I power down, and I insert a new hard drive in place of the old one as mentioned in all the documentation (same SATA) cable. 4. UnRaid rightfully complains that I accidentally inserted a hard drive greater than the parity hard drive (parity was 750GB, I inserted a 1TB drive). I shut down the drive. 5. I go out and buy a new hard drive (same size as the failed drive 500GB) and insert it inside and boot up. 6. I notice that the UnRaid server has problems getting to the regular interface. I noticed that it sometimes says 'mounting' and is freezing. I try several times to no avail. Now if I insert the new drive, the computer gets the root login, but the keyboard is frozen, and you cannot access the web interface. 7. I remove the replacement drive, and it boots up fine. However, it now complains that the replacement drive is missing (which I'm worried about, since it might imply that my original drive is not longer there) 8. I may have clicked on the 'restore' option, at one point, to see if that helped, but I do not believe it did. However, I have not brought the array online since. Normally according to documentation an option to start up and 'rebuild' the array should appear, but I never got it. I wonder if inserting the larger drive size caused an untested case to occur. 9. As it stands now, I have the option of just starting up the system again, but it seems like if I do that, i'll detect all the hard drives as new, and reset the existing parity. I have not started up the array where parity sync runs yet, so I think all the data is intact. (not sure here) Ideally, I wish I could just 1. Get to the point where the system has 1 failed drive, so the system is unprotected, but it runs of parity information so I can access the data. 2. Get to the point where inserting a new drive to replace the old drive doesn't cause the system to freeze. As luck has it, I lost my most important drive with all my old files. Any help would be greatly appreciated. Thanks. -Daniel
July 25, 200916 yr There are warnings everywhere you look in the forums and wiki that you NOT press the restore button. It is horribly named, but unfortunately has not been a priority for the author to change. If you pressed restore (and since you mention I'm assuming you did), your parity data, although "correct", is NOT associated with your array. If you start the array with the good drives - or with the good drives + the replacement drive - unRAID will overwrite the parity data and any hope of getting your failed disk contents back will be lost. 1. Get to the point where the system has 1 failed drive, so the system is unprotected, but it runs of parity information so I can access the data. The only possible way of getting back to here is to replace the super.dat file with the backup one. I'll leave to Joe L. to confirm if this is a good option. I have never tried it but he may have. 2. Get to the point where inserting a new drive to replace the old drive doesn't cause the system to freeze. As luck has it, I lost my most important drive with all my old files. This is possible. First, however, you have to get your replacement disk to be recognized without freezing the server. I wonder if maybe your data cable or power cable / connection are bad or lose. Once you get that fixed, you should be able to use a modified "trust my parity" procedure to rebuild it. Look at this post which give instructions. Sorry I have to run this morning. Will post back more detailed instructions later if someone else has not already done so. Say calm and don't do anything in the heat of the moment. Make sure you understand clearly the series of steps you need to follow before starting.
July 25, 200916 yr The only possible way of getting back to here is to replace the super.dat file with the backup one. I'll leave to Joe L. to confirm if this is a good option. I have never tried it but he may have. I agree, on your flash drive, in the config folder will be a super.old file. It was the super.dat file before you pressed the "restore" button. (Which does NOT restore data, but an initial configuration based on currently working/assigned drives, but without valid parity. If you start the array after you press it, you will overwrite your old parity, and any data on a failed drive) In one respect, you are very lucky you did not start the array after you pressed restore. You can probably copy config/super.old to overwrite config/super.dat. Do this on a windows PC, in fact, make a copy of the entire "config" folder while you have it there. I too have a commitment that will keep me busy the bulk of the day. So I'll check in later, but DO NOT START THE ARRAY unless using a modified version of the "trust" my disk procedure. Your first priority is to get the array stable to where it is showing the failed disk as missing and or disabled. (It really does not matter which) Cabling is the most likely issue, with loose connectors a huge problem. Then, with the drive missing we can proceed. Also, please specify the exact version of unRAID you are using. And attach a copy of your syslog once you get stable. Again, DO NOT START THE ARRAY unless you are back to where only the failed drive is showing as missing/disabled. Joe L.
July 25, 200916 yr Author Thanks bjp999 and Joe L., I greatly appreciate how fast all of you joined this thread to try to help me out. As it stands, it seems like the consensus is to try to get back to the point where the it shows the missing drive again. This can be done by: 1. Ensuring that all the cables are hooked up correctly. 2. Try to replace the existing config with the old one. If the old one does not show up and I am only able to get it to load without freezing then I will need to use commands outlined here to ask the raid array to rebuild the array. (I'll save this one as a last resort). I'm currently using 4.3.3. UPDATE: Looks like a damaged cable might have been it, after replacing the cable, unRaid now detects the new replacement drive. However, all hard drives show up as blue (new). (Note the original drive actually failed, not due to the a loose cable. It was making loud clicking noises and was producing way less heat than my other hard drive) Currently, it detects the new hard drive and prints the serial number, there's no reference to the old one. I guess my next step is to try replacing the config file with the older back up one and go from there. I'll wait till later tonight to give it a try. Thanks again Joe and bjp999 for all your help. -Daniel
July 25, 200916 yr Look at the date/time stamp on the super.old file. If it is very recent, it is probably messed up to. I searched and found anothe thread that might be better than the one I previously posted to help another user recover a disk after he pressed restore (or something similar). Please ask questions if not clear. http://lime-technology.com/forum/index.php?topic=3541.msg30699#msg30699 Note that hopefully after rebuilding the disk the disk will be immediately readable. The final few steps dealing with running reiserfsck should not be necessary.
July 25, 200916 yr Look at the date/time stamp on the super.old file. If it is very recent, it is probably messed up to. I searched and found anothe thread that might be better than the one I previously posted to help another user recover a disk after he pressed restore (or something similar). Please ask questions if not clear. http://lime-technology.com/forum/index.php?topic=3541.msg30699#msg30699 Note that hopefully after rebuilding the disk the disk will be immediately readable. The final few steps dealing with running reiserfsck should not be necessary. Even while rebuillding the disk it will be immediately available via the parity disk and all the other data disks. If you see any evidence of the parity disk being written to instead the failed disk, stop the array immediately. (you might see a few writes to it as disks get mounted, but for th emost part you should see lots of writes to the replacement you installed to replace the failed disk) After invoking the mdcmd set invalidslot command for the FAILED drive, you should see a confirmation. If you do not see it, DO NOT proceed. Remember, the number you supply to the set invalid slot command most correspond to the FAILED disk. If it was disk 2 that failed, you would use "mdcmd set invalidslot 2", if disk1, then it would be "mdcmd set invalidslot 1" That command MUST be entered at a command prompt with you in the home directory of the "root" user. To be certain, you can always type cd first. The "mdcmd set invalidslot" command must be typed AFTER you press the button labeled "restore" and BEFORE you press "Start" If you have ANY questions, ASK first. Joe L.
July 26, 200916 yr Author Just tried the steps. Set the disk to invalid, got the confirmation msg that it worked and then I started the array. What happenened is that it says that it was 'mounting', but after I get no response after waiting a bit. I try the 'refresh' button, and it cannot access the web interface any more. I also noticed that the keyboard that I had typed in the mdcmd earlier isn't responding. This seems like the same thing that happened when the system first tries to restore the drive automatically. Hmm... Waited about 10 mintues now, no response from server. -Daniel
July 26, 200916 yr Author Update, from your last tips on freezing, I replaced 3 sata cables that I think may have had an issue, and as it booted up, I think it's restoring it My disk3 (the failed disk) is recovering right now, and all the other disks are just doing 'reads'. 50 writes to the parity. Any how, I’ll give you guys an update after 555 minutes (recovery time). I can access other drives at this time, but disk3 is not accessible at this time. Thanks for all your help! -Daniel
July 26, 200916 yr Author Update: Looks like the rebuild completed. However, under the "Free" Column, it says "Unformated" (probably since the replacement disk was new) There's a message: "Unformatted disk(s) present" and a button FORMAT "Format will create a file system in all unformatted disks." I wonder if this button actually formats the hard drive or just creates the file system tables. Hmm.. Usually there's a warning if it deletes all my data, but I thought I'd ask just in case. Thanks.
July 26, 200916 yr Author Ok, found a thread that has a similar situation. http://lime-technology.com/forum/index.php?topic=3716.msg32410#msg32410 It says to post the results of 1. fdisk -l /dev/sd? and 2. vol_id /dev/sd?1
July 26, 200916 yr Update: Looks like the rebuild completed. However, under the "Free" Column, it says "Unformated" (probably since the replacement disk was new) There's a message: "Unformatted disk(s) present" and a button FORMAT "Format will create a file system in all unformatted disks." I wonder if this button actually formats the hard drive or just creates the file system tables. Hmm.. Usually there's a warning if it deletes all my data, but I thought I'd ask just in case. Thanks. DO NOT PRESS THE FORMAT BUTTON OR YOU WILL LOSE YOUR DATA. You will force it to erase it. even in the parity drive.. The "Unformatted" probably indicates that the disk could not be mounted. It probably has some file-system corruption. Post another syslog, but DO NOT press format. Do not do anything till RobJ can also look at the syslog. Joe L.
July 26, 200916 yr Ok, found a thread that has a similar situation. http://lime-technology.com/forum/index.php?topic=3716.msg32410#msg32410 It says to post the results of 1. fdisk -l /dev/sd? and 2. vol_id /dev/sd?1 WOW. Look at the temperatures on your drives!!! You need to get some cooling to get the temps under 50, and perferably below 45. It is URGENT that you get that taken care of first, otherwise other drives are going to start failing. I have NEVER seen a drive at 59C.
July 26, 200916 yr Author Thanks. The case is wide open right now and the room is cool. I think all the reads used for the recovery for the last 8 hours caused the temperature to skyrocket. I just hit the 'spin down' all drives temperarily to help reduce the temperature. (hopefully this wasn't a stupid idea) Results of those commands: root@Tower:~# fdisk -l /dev/sd3 root@Tower:~# vol_id /dev/sd31 /dev/sd31: error opening volume root@Tower:~# Error log attached.
July 26, 200916 yr Update: Looks like the rebuild completed. However, under the "Free" Column, it says "Unformated" (probably since the replacement disk was new) There's a message: "Unformatted disk(s) present" and a button FORMAT "Format will create a file system in all unformatted disks." I wonder if this button actually formats the hard drive or just creates the file system tables. Hmm.. Usually there's a warning if it deletes all my data, but I thought I'd ask just in case. Thanks. DO NOT PRESS THE FORMAT BUTTON OR YOU WILL LOSE YOUR DATA. You will force it to erase it. even in the parity drive.. The "Unformatted" probably indicates that the disk could not be mounted. It probably has some file-system corruption. Post another syslog, but DO NOT press format. Do not do anything till RobJ can also look at the syslog. Joe L. One fact about unRAID that regularly bites users on the butt when they have a drive failure. That fact is that parity can be totally screwed up and the array will run ABSOLUTELY NORMALLY. You will not get errorrs in the syslog, you will not get errors on the Web GUI, your data will be read/written correctly, and your performance will be as expected. The ONLY way to know that parity is being maintained is to run a parity check. It is especially important to run a parity check after building the parity. A SUCCESSFUL PARITY BUILD DOES NOT MEAN THAT PARITY IS CORRECT! There have been numerous incompatibility issues that result in the parity build appearing to run successfully whilte parity is in fact NOT correct. I very much hope that your parity was correct at the time of failure. But if it were and based on the steps described, you should have recovered. But you may have done other things, and if so chances are good that staying calm and folliwing instructions will result in a recovery. But if parity was never correct, the data will not recover.
July 26, 200916 yr Ok, found a thread that has a similar situation. http://lime-technology.com/forum/index.php?topic=3716.msg32410#msg32410 It says to post the results of 1. fdisk -l /dev/sd? and 2. vol_id /dev/sd?1 WOW. Look at the temperatures on your drives!!! You need to get some cooling to get the temps under 50, and perferably below 45. It is URGENT that you get that taken care of first, otherwise other drives are going to start failing. I have NEVER seen a drive at 59C. 59C = 138.2 F Way too hot for comfort!!
July 26, 200916 yr The correct device name for t he vol_id command would not be sd31. For the first partition it is always a three letter designation followed by the partition number. sda1, sdb1, sdc1,sdn1, etc.
July 26, 200916 yr Author Thanks for all the help thus far Joe. (Re: do not format) RE: Temperature. Woah! I just discovered that the 120mm case fan that normally cools 4 hard drives has stopped working (this probably explains why the original hard drive failed in the first place).
July 26, 200916 yr Author Ok, temperature is lower now. Most hard drives are at 35 the hotest one is at 44. root@Tower:/dev# ls sd* sda sda1 sdb sdb1 sdc sdc1 sdd sdd1 sde sde1 sdf sdf1 sdg sdg1 sdh sdh1 root@Tower:/dev# vol_id /dev/sda1 ID_FS_USAGE=filesystem ID_FS_TYPE=ntfs ID_FS_VERSION= ID_FS_UUID=56309EA4309E8B1F ID_FS_UUID_SAFE=56309EA4309E8B1F ID_FS_LABEL= ID_FS_LABEL_SAFE= root@Tower:/dev# vol_id /dev/sdb1 ID_FS_USAGE=filesystem ID_FS_TYPE=ntfs ID_FS_VERSION= ID_FS_UUID=56309EA4309E8B1F ID_FS_UUID_SAFE=56309EA4309E8B1F ID_FS_LABEL= ID_FS_LABEL_SAFE= root@Tower:/dev# vol_id /dev/sdc1 ID_FS_USAGE=filesystem ID_FS_TYPE=reiserfs ID_FS_VERSION=3.6 ID_FS_UUID=2292354a-9679-46c6-b13d-096180e93f52 ID_FS_UUID_SAFE=2292354a-9679-46c6-b13d-096180e93f52 ID_FS_LABEL= ID_FS_LABEL_SAFE= root@Tower:/dev# vol_id /dev/sdd1 ID_FS_USAGE=filesystem ID_FS_TYPE=reiserfs ID_FS_VERSION=3.6 ID_FS_UUID=d38f1d8b-e613-44f4-aaa2-d2fcc093ae91 ID_FS_UUID_SAFE=d38f1d8b-e613-44f4-aaa2-d2fcc093ae91 ID_FS_LABEL= ID_FS_LABEL_SAFE= root@Tower:/dev# vol_id /dev/sde1 ID_FS_USAGE=filesystem ID_FS_TYPE=reiserfs ID_FS_VERSION=3.6 ID_FS_UUID=f56e2954-deee-4861-80a2-95af3f9b3182 ID_FS_UUID_SAFE=f56e2954-deee-4861-80a2-95af3f9b3182 ID_FS_LABEL= ID_FS_LABEL_SAFE= root@Tower:/dev# vol_id /dev/sdf1 ID_FS_USAGE=filesystem ID_FS_TYPE=reiserfs ID_FS_VERSION=3.6 ID_FS_UUID=585c4239-dbb6-4c80-95d7-b7bf0747bae9 ID_FS_UUID_SAFE=585c4239-dbb6-4c80-95d7-b7bf0747bae9 ID_FS_LABEL= ID_FS_LABEL_SAFE= root@Tower:/dev# vol_id /dev/sdg1 ID_FS_USAGE=filesystem ID_FS_TYPE=reiserfs ID_FS_VERSION=3.6 ID_FS_UUID=2f30b534-f2b8-458b-a317-415d6ce3d4ea ID_FS_UUID_SAFE=2f30b534-f2b8-458b-a317-415d6ce3d4ea ID_FS_LABEL= ID_FS_LABEL_SAFE= root@Tower:/dev# vol_id /dev/sdh1 ID_FS_USAGE=filesystem ID_FS_TYPE=vfat ID_FS_VERSION=FAT32 ID_FS_UUID=501F-D707 ID_FS_UUID_SAFE=501F-D707 ID_FS_LABEL=UNRAID ID_FS_LABEL_SAFE=UNRAID root@Tower:/dev#
July 26, 200916 yr now, post the copy of the syslog, so we can figure out what is really going on. Do not press "format" Do not press "restore" I would not do any writes to the array either at this time... Do type this: mount to let us see what is actually mounted. Joe L.
July 26, 200916 yr Author Thanks Joe. root@Tower:/dev# mount proc on /proc type proc (rw) sysfs on /sys type sysfs (rw) usbfs on /proc/bus/usb type usbfs (rw) /dev/sdh1 on /boot type vfat (rw) fusectl on /sys/fs/fuse/connections type fusectl (rw) /dev/md2 on /mnt/disk2 type reiserfs (rw,noatime,nodiratime) /dev/md5 on /mnt/disk5 type reiserfs (rw,noatime,nodiratime) /dev/md6 on /mnt/disk6 type reiserfs (rw,noatime,nodiratime) /dev/md1 on /mnt/disk1 type reiserfs (rw,noatime,nodiratime) /dev/md7 on /mnt/disk7 type reiserfs (rw,noatime,nodiratime) /dev/md4 on /mnt/disk4 type reiserfs (rw,noatime,nodiratime) My first dump of the system log can be found in my previous message: http://lime-technology.com/forum/index.php?topic=4073.msg36015#msg36015 http://lime-technology.com/forum/index.php?action=dlattach;topic=4073.0;attach=1751 However, I just did another one in case you wanted the most recent one. Please see the attachment. Thanks again Joe!
July 26, 200916 yr There are several issues I can see quickly. First, your flash drive has some file-system corruption of its own as evidenced by these lines in the syslog: Jul 26 00:52:58 Tower kernel: FAT: Filesystem panic (dev sdh1) Jul 26 00:52:58 Tower kernel: fat_free_clusters: deleting FAT entry beyond EOF Jul 26 00:52:58 Tower kernel: File system has been set read-only The fact that the file-system is set to read-only means that changes to the config/system.dat file cannot be saved. This is not good. You probably can fix this problem by using "chkdisk" on your windows PC on the flash drive. Once it is fixed, then at least the corrective actions you take in getting your data drive recovered will be reflected in the true array status. If not fixed, the array state cannot be saved. Then, further on in the syslog, the file-system on /dev/md3 cannot be mounted. This is why it appears unformatted on the unRAID management display... The specific lines in the syslog are: As a result of this attempt to mount disk3: Jul 26 00:52:58 Tower emhttp: shcmd (20): mount -t reiserfs -o noatime,nodiratime /dev/md3 /mnt/disk3 >/dev/null 2>&1 You got this error: Jul 26 00:52:59 Tower kernel: ReiserFS: md3: warning: sh-2021: reiserfs_fill_super: can not find reiserfs on md3 Jul 26 00:52:59 Tower emhttp: shcmd: shcmd (20): exit status: 32 Jul 26 00:52:59 Tower emhttp: shcmd (21): rmdir /mnt/disk3 Can you tell me which /dev/sd? device or physical disk you have assigned to disk3 on your disk assignment page? (I'm not sure I know enough to determine that from the syslog alone) Joe L.
July 26, 200916 yr Oh yes, in this state, with a drive not mounted, DO NOT press the "Check Parity" button. If you do, you would overwrite what used to be in parity tracking drive3 with zeros, effectively erasing all traces of drive3. Probably best to not write any new files to the array for now. Joe L.
July 26, 200916 yr Author Thanks for the warning on parity-check and new files Joe. (I also had a feeling about this, so I didn't do either on purpose) Disk 3 is the Seagate 500GB Drive pci-0000:00:0a.0-scsi-0:0:0:0 (sda) ata-ST3500418AS_5VM180JB If I run chkdisk on the flash drive, i'll need to turn off the server first, unless the flash disk is hot swappable? I haven't turned off the system since it attempted to recover data from last night. Thanks, -Daniel
July 26, 200916 yr Thanks for the warning on parity-check and new files Joe. (I also had a feeling about this, so I didn't do either on purpose) Disk 3 is the Seagate 500GB Drive pci-0000:00:0a.0-scsi-0:0:0:0 (sda) ata-ST3500418AS_5VM180JB If I run chkdisk on the flash drive, i'll need to turn off the server first, unless the flash disk is hot swappable? I haven't turned off the system since it attempted to recover data from last night. Thanks, -Daniel Its probably safe to try to unplug the USB flash drive. Normally you would never, since the unRAID server needs to be able to write to it, but since it is read-only now, can't hurt to try. You can first un-mount it, then un-plug it. umount /boot Then fix it, (Don't re-format it, just run chkdisk, or scandisk), then plug it back in to the unRAID server. I'm pretty sure it will mount itself. You'll know if you type ls /boot and see the files there. Once that issue is resolved, we can attempt to run a file system check on disk3 Joe L.
Archived
This topic is now archived and is closed to further replies.