February 11, 201214 yr I regularly check my unraid for errors and today I end up with this (see attachement). I have 2 drives that had errors (one had 49 errors and the other 14).. both drives show a temperature of 0*C. The one with 49 errors has a green marker and the one with 14 errors has a red marker. Not sure if I should've done this but I stopped the array and started it again. Now both those drives show as unformatted. I know that 2 bad drives in an array is normally pretty bad. So I'm asking, what would be my next step to recover the data? Anyway to tell unraid that one of the drive is fine so I recover that one and then the other? I'm just not sure what to do from here on...
February 12, 201214 yr http://lime-technology.com/forum/index.php?topic=9880.0 http://lime-technology.com/forum/index.php?topic=9822.0
February 12, 201214 yr Author Yea... forgot to attach the syslog. I tried running smartctl but if I type "smartctl -a -d ata /dev/sda" it gives me "Smartctl: Device Read Identity Failed (not an ATA/ATAPI device)" I then tried "smartctl -a -A /dev/sda" but it gives me "Device: /3:0:0:0 Version: scsiModePageOffset: response length too short, resp_len=47 offset=50 bd_len=46" Humm, it gives me these kind of errors for SDA and SDB... the 2 drives giving me issues. The computer is in a closet.. but nonetheless I went and checked the connections. Everything was OK. I find it weird that all of a sudden, read errors happened on 2 seperate drives.. should I just try shutting down the machine and start it again? Maybe the HD controller spazzed out? syslog.txt
February 12, 201214 yr Yea... forgot to attach the syslog. I tried running smartctl but if I type "smartctl -a -d ata /dev/sda" it gives me "Smartctl: Device Read Identity Failed (not an ATA/ATAPI device)" I then tried "smartctl -a -A /dev/sda" but it gives me "Device: /3:0:0:0 Version: scsiModePageOffset: response length too short, resp_len=47 offset=50 bd_len=46" Humm, it gives me these kind of errors for SDA and SDB... the 2 drives giving me issues. The computer is in a closet.. but nonetheless I went and checked the connections. Everything was OK. I find it weird that all of a sudden, read errors happened on 2 seperate drives.. should I just try shutting down the machine and start it again? Maybe the HD controller spazzed out? I would shutdown the machine. Power off , remove power cord for 30 seconds, Then plug in and restart. No matter what. DO NOT press the FORMAT button if they were already formatted before the failure.
February 12, 201214 yr Author Ok, so I shutdown the machine, removed power cord, waited and started it up again.. I heard one of the drive ticking (normally an HD Ticks once but here I'm talking multiple ticks). It stopped after ~10 ticks or so... I waited out but unraid never got online (couldn't ping the machine). So I decided to shut it down and unplug the ticking hard drive (of course it was one of the faulty drive, the one with the most errors). Now unraid gets online but of course its saying I'm missing a hard drive and the other one is still a red dot but now has a temperature reading. I attached the syslog from this bootup. I also attached the smart report of the drive that has a red dot now that it responds to it. I'm thinking the failing drive was messing up with the data from this drive, hence why unraid saw errors from it as well. So I'm thinking, is there a way to tell unraid that this drive is fine so I can swap out the failed drive and rebuild it? syslog.txt smart.txt
February 12, 201214 yr So I'm thinking, is there a way to tell unraid that this drive is fine so I can swap out the failed drive and rebuild it? There might be, but I do not know how at this point. Does this drive say Unformatted?
February 12, 201214 yr Author Can't start the array, Too many wrong and/or missing disks! So here's my plan: Change the failing hard drive Follow instructions at http://lime-technology.com/wiki/index.php?title=Make_unRAID_Trust_the_Parity_Drive,_Avoid_Rebuilding_Parity_Unnecessarily Start the array, this should rebuild the failing/new drive and leave the other hd untouched... unless I'm wrong and there was actual errors on the drive. What do you think?
February 12, 201214 yr you should not do this..... see if you can replace one at a time. if you do his the files on the bad drive will be lost forever.
February 12, 201214 yr Author Well that is the thing, let me state the facts again: - Unraid sees two drives with errors - One of the drives completely failed and needs to be replaced - The other drive is actually fine and they were false errors. But since unraid still sees it as bad, I can't just rebuild the failed drive. - There MUST be a way for me to tell unraid that the drive with the false errors is fine. Then I could rebuild the drive that failed.
February 12, 201214 yr The plan may work and I don't think you have any other options. What you need to first do is get the drive properly partitioned once before rebuilding since unRAID will only rebuild the data on the partition and not the whole drive. Likely, the best plan would be to make a new temp USB stick and boot that just so you can assign the new drive as a data drive and start the array once. You can format it too if you like but that's not necessary. You would then use the trust my parity procedure but instead of using 99 you would use the number of the disk you want rebuilt, in this case 1. This command basically tells the unRAID whick drive to rebuild so the initconfig will reset everything to be good to start and then the set invalidslot command will tell unRAID to rebuild the partition on disk1 when the start button is pressed. The reason the disk needs the partition is that the partition command is normally given by the interface code before starting the rebuild but you have bypassed this. You will likely have to run reiserfsck and do some fixing on both disks once they are back green and it's still possible you will not get back your data. Be careful because it you do something wrong you can hose another disk. Peter
February 12, 201214 yr Author Thank you for the instructions, I know how to fiddle with linux but I'm far from an expert. The only thing I'm not sure about is the partitioning of the drive. I'll boot up with a seperate key to see what it does, maybe its gonna be more clear at that point but I'm a bit scared of starting the array from that temp usb key. Wouldn't there be a way through shell to partition the drive? Or is it a bit more complicated than that?
February 12, 201214 yr Author Alright, so far so good... I did the temp key thing to partition the new drive. Then I booted up with the real key. Made sure every drive was assigned correctly and then did the "initconfig" command in telnet. All drives went blue. I then did "mdcmd set invalidslot 1" but that command didn't output anything, the instructions say it should output: cmdOper=set cmdResult=ok But it didn't. Did it execute correctly? Anyway I can check? syslog maybe? Just want to make sure before I start the array. Here's whats in my syslog from the initconfig and invalidslot commands: Feb 12 14:03:40 Tower login[1660]: ROOT LOGIN on `pts/0' from `10.0.0.250' Feb 12 14:04:22 Tower emhttp: shcmd (24): modprobe -rw md-mod 2>&1 | logger Feb 12 14:04:22 Tower emhttp: shcmd (25): modprobe md-mod super=/boot/config/super.dat slots=8,32,8,16,8,0,8,48,8,64,3,64 2>&1 | logger Feb 12 14:04:22 Tower kernel: md: unRAID driver removed Feb 12 14:04:22 Tower kernel: xor: automatically using best checksumming function: pIII_sse Feb 12 14:04:22 Tower kernel: pIII_sse : 5518.800 MB/sec Feb 12 14:04:22 Tower kernel: xor: using function: pIII_sse (5518.800 MB/sec) Feb 12 14:04:22 Tower kernel: md: unRAID driver 1.1.1 installed Feb 12 14:04:22 Tower kernel: read_file: error 2 opening /boot/config/super.dat Feb 12 14:04:22 Tower kernel: md: could not read superblock from /boot/config/super.dat Feb 12 14:04:22 Tower kernel: md: initializing superblock Feb 12 14:04:22 Tower kernel: md: import disk0: [8,32] (sdc) WDC WD20EARS-00M WD-WMAZA0621109 size: 1953514552 Feb 12 14:04:22 Tower kernel: md: disk0 new disk Feb 12 14:04:22 Tower kernel: md: import disk1: [8,16] (sdb) WDC WD20EARX-00P WD-WCAZAD229351 size: 1953514552 Feb 12 14:04:22 Tower kernel: md: disk1 new disk Feb 12 14:04:22 Tower kernel: md: import disk2: [8,0] (sda) WDC WD20EARS-00M WD-WMAZA0723835 size: 1953513492 Feb 12 14:04:22 Tower kernel: md: disk2 new disk Feb 12 14:04:22 Tower kernel: md: import disk3: [8,48] (sdd) WDC WD10EADS-00M WD-WCAV50196695 size: 976762552 Feb 12 14:04:22 Tower kernel: md: disk3 new disk Feb 12 14:04:22 Tower kernel: md: import disk4: [8,64] (sde) ST31000340AS 9QJ0J93X size: 976762552 Feb 12 14:04:22 Tower kernel: md: disk4 new disk Feb 12 14:04:22 Tower kernel: md: import disk5: [3,64] (hdb) ST3750640A 5QD4ZKA9 size: 732574552 Feb 12 14:04:22 Tower kernel: md: disk5 new disk Feb 12 14:04:22 Tower kernel: mdcmd (1): set md_num_stripes 1280 Feb 12 14:04:22 Tower kernel: mdcmd (2): set md_write_limit 768 Feb 12 14:04:22 Tower kernel: mdcmd (3): set md_sync_window 288 Feb 12 14:04:22 Tower kernel: mdcmd (4): set spinup_group 0 0 Feb 12 14:04:22 Tower kernel: mdcmd (5): set spinup_group 1 4 Feb 12 14:04:22 Tower kernel: mdcmd (6): set spinup_group 2 2 Feb 12 14:04:22 Tower kernel: mdcmd (7): set spinup_group 3 0 Feb 12 14:04:22 Tower kernel: mdcmd (: set spinup_group 4 0 Feb 12 14:04:22 Tower kernel: mdcmd (9): set spinup_group 5 0 Feb 12 14:05:22 Tower kernel: mdcmd (10): set invalidslot 1 Peter or somebody else, can you confirm that I'm good to go (or at least the best bet at recovering the array)
February 12, 201214 yr It's showing the command was successful in the syslog but I still believe you should have gotten those responses. You could try the command again. You will have to just try it if there is no response and you know you put the command in correctly. The only way to truely know is to start the array. disk1 should show writes and all the others should show reads. The problem with starting if it didn't work is that the parity will be written and you'll cause damage that likely can't be undone. Be quick with the start button and the refreshes to monitor the disks and stop it quickly if it seems wrong. Peter
February 12, 201214 yr Author Humm.. yea running the command again didn't return a response but it showed up in the log again. No disrespect towards you Peter, you've been very helpful, but what are the chances of a developper/support of lime tech chimming in on the subject? I know we're in the weekend but I don't mind waiting till tomorow/tuesday before trying to start it.
February 12, 201214 yr Humm.. yea running the command again didn't return a response but it showed up in the log again. No disrespect towards you Peter, you've been very helpful, but what are the chances of a developper/support of lime tech chimming in on the subject? I know we're in the weekend but I don't mind waiting till tomorow/tuesday before trying to start it. Contact lime-tech directly. http://www.lime-technology.com/company/contact Point him to the thread.
February 13, 201214 yr The chances of seeing suport from Limetech here is nil. It hardly ever happens. You can try via email and see what answers you get. Peter
February 13, 201214 yr Let us know if he hasn't responded in a few days. I know they were moving offices from midwest to the west coast.
February 14, 201214 yr Author Good to know. They haven't responded yet. I'll let you all know as soon as they do.
February 14, 201214 yr Humm.. yea running the command again didn't return a response but it showed up in the log again. No disrespect towards you Peter, you've been very helpful, but what are the chances of a developper/support of lime tech chimming in on the subject? I know we're in the weekend but I don't mind waiting till tomorow/tuesday before trying to start it. The mdcmd command has changed over the years. It used to respond with OK. It no longer does. If using the newer beta versions of unRAID, do NOT refresh the status page. Just press "Start" If you see it writing to the parity drive, press 'Stop" as fast as you can. It should be writing to disk1, and reading from all the others.
February 14, 201214 yr Humm.. yea running the command again didn't return a response but it showed up in the log again. No disrespect towards you Peter, you've been very helpful, but what are the chances of a developper/support of lime tech chimming in on the subject? I know we're in the weekend but I don't mind waiting till tomorow/tuesday before trying to start it. The mdcmd command has changed over the years. It used to respond with OK. It no longer does. If using the newer beta versions of unRAID, do NOT refresh the status page. Just press "Start" If you see it writing to the parity drive, press 'Stop" as fast as you can. It should be writing to disk1, and reading from all the others. The OP is running 4.6. As far as I know, the command should give a response up to version 4.7. It did indicate it worked in the syslog though. Peter
February 14, 201214 yr Humm.. yea running the command again didn't return a response but it showed up in the log again. No disrespect towards you Peter, you've been very helpful, but what are the chances of a developper/support of lime tech chimming in on the subject? I know we're in the weekend but I don't mind waiting till tomorow/tuesday before trying to start it. The mdcmd command has changed over the years. It used to respond with OK. It no longer does. If using the newer beta versions of unRAID, do NOT refresh the status page. Just press "Start" If you see it writing to the parity drive, press 'Stop" as fast as you can. It should be writing to disk1, and reading from all the others. The OP is running 4.6. As far as I know, the command should give a response up to version 4.7. It did indicate it worked in the syslog though. Peter It does not give a response in 4.7. I do not remember which version it stopped.
February 14, 201214 yr Author I was using 4.6 but due to some mishap, had to upgrade to 4.7 while doing those commands. Alright, so I did start it, was a bit scared when I saw writes to the parity and was unable to stop it (refreshing was giving me "Starting...." with only a refresh button available) but I assume that was due to me putting in a bigger drive than the failed one. So anyway, after a few mins, it started the parity check and from then on, only wrote to the new drive, reading from the others. It completed a few mins ago and now it says the parity is valid. So at this point, I think to make sure everything is really good, I should do a full parity check?
Archived
This topic is now archived and is closed to further replies.