I replaced a failing 500gb sata drive today with a new 750gb WD Green drive, and during a parity rebuild, the machine appears to have locked up (the screen was filled with text about not being able to shrink a mounted file system).  After letting it run for several hours (with no disk activity at all on the box), I restarted.  Now when I bring the system up, the initial screen shows the that the system is in the Initial configuration mode.  Since it wasn't finished with the parity sync when it crashed (and I'm STILL not sure why, the new drive looks fine via unmenu's smart status, no errors or anything).  At this point, it's giving me 2 options, start and build parity (I'm guessing I'll loose everything off the 500g that died, since parity is flagged bad now?), and restore, all drives marked as new, but all data is unaffected.  Since I'm using a fairly high split level on my shares, I have NO idea what the contents of the 500g that died are (and no way to get 'em back either), what are my alternatives, and which choice should I make?

I replaced a failing 500gb sata drive today with a new 750gb WD Green drive, and during a parity rebuild, the machine appears to have locked up (the screen was filled with text about not being able to shrink a mounted file system).  After letting it run for several hours (with no disk activity at all on the box), I restarted.  Now when I bring the system up, the initial screen shows the that the system is in the Initial configuration mode.  Since it wasn't finished with the parity sync when it crashed (and I'm STILL not sure why, the new drive looks fine via unmenu's smart status, no errors or anything).  At this point, it's giving me 2 options, start and build parity (I'm guessing I'll loose everything off the 500g that died, since parity is flagged bad now?), and restore, all drives marked as new, but all data is unaffected.  Since I'm using a fairly high split level on my shares, I have NO idea what the contents of the 500g that died are (and no way to get 'em back either), what are my alternatives, and which choice should I make?

Before you do anything, post a syslog.  Hopefully it will have a clue to what is happening.    Until you hear otherwise, do not press the dreaded "restore" button.

 

You might want to try running reiserfsck on the new drive.  (You will need to do this from the command line) The rebuild process might have gotten far enough along in the rebuild process where the the file-system exists and some files are in place. 

 

Joe L.

BE VERY CAREFUL!

 

Do not start your array.

 

You will likely be able to recover.

 

Will post back in a few with instructions.

You have 2 issues.

 

1.  Somehow, in hooking ou the new drive, you have created a problem.  It is likely a bad cable or bad drive.  I would recommend running a smartctl report on the new drive and seeing if there are reallocated sectors or pending sectors.  If there are, the drive may be bad (it happens rather frequently that a new drive is defective).  If there are not attribute errors, but there are a bunch of errors on the drive (ata errors), it will point towards a bad cable (or some other connection problem - which could include a bad backplane / docking device, a bad or lost SATA plug on the MB or controller).  At a minimum I would replace the data cable to the drive with a fresh cable.  Eliminate backplanes / docks if you can.  You may also want to connect a different power cable to the drive.  It is also possible that you jiggled a connector to another drive and that the new drive is fine.  Double check all connectors are tightly in place.  It may take some trial and error to figure out what is wrong.  The only way to "try" anything, though, is to follow the procedure below.

 

2.  The second problem is not a mystery.  unRAID has forgotten your array configuration.  Telling it to create a new one will cause data loss (as you suspected).  What you need is a way to manually define your array and tell unRAID to rebuild your failed drive.  Fortunately there is a way to do that.  It hinges on one very imporant thing - do you know which drive is parity?

 

Actually the easiest way to recover is if you have a recent backup of your config directory.  If you do, you can restore that and reboot unRAID and it would come up and try to rebuild your drive.  The following assumes you do not have a backup.

 

When you say "Now when I bring the system up, the initial screen shows the that the system is in the Initial configuration mode" I assume that means that no drives are assigned to any slots.  If that is not the case, and your drives are already assigned and have blue dots next to them, that may save you a few steps below.

 

First thing you should do is press the restore button.  It is likely unnecesssary, but at least you know everything is reset.

 

Go to the devices page and make sure to assign the parity drive to the parity slot.  GET THIS RIGHT!

 

Assign your data disks to the data slot.  It is not important to unRAID that the right disk be placed in the right slot, you just need to assign them all.  (It may matter to you though, so putting the right drives in the right slots is probably a good idea.)  Take special note of which slot you assign the NEW 750G drive.

 

Go to the main page and double, triple, and quadruple check that you have all of your drives assigned, and that parity is in the parity slot.

 

Go to a telnet window and enter this command:

 

mdcmd set invalidslot X

 

where X is the slot where the new 750G drive is located.  For example, if it is in slot 3 (i.e., disk3), the command will be "mdcmd set invalidslot 3".  Do not prefix the number with a 0!  Before hitting enter, make sure you've got this command right.  Then hit enter.

 

(This command tells unRAID that when you start the array, it should rebuild that drive.  The default is "0" meaning rebuild parity.  You do NOT want to rebuild parity, you want to rebuild your data drive).

 

You can now go to the Web GUI and start the array.  It should start to rebuild the 750G drive (you'll should be able to see the numbers of reads and writes to each drive.  The new 750G drive should be getting the lion's share of the writes.)

 

If you didn't fix problem 1, this will likely fail again.  The good news is that you can repeat this process if it crashes again.

 

Immediately after starting the array, save your syslog for future posting.  Then, open a telnet session and run the command "tail -f /var/log/syslog" on a workstation.  If the system crashes, you will have the most recent log errors which you can post.  These may be able to help you diagnose what is wrong with the drive.  Of course if the system stops rebuilding but you are still able to logon to the box, you should capture complete syslog and post it.  That would be best if you have another problem.

 

Good luck  Take your time.  Don't make mistakes!

 

The array configuration shows all the drives in their correct slots, it just shows them with a blue dot.  I rebooted the system before I went to bed last night (and didn't stay up to watch), but I noticed just now that the 'new' drive isn't showing up at all now (shows Not Installed).  that particular drive is in my 4x3, in the 3rd slot (where the 500g was), and shows power, but I did notice when I put it in the slot and closed the door it didn't click in like the other drives, just felt different (hard to describe, but it's definitely not the same).  At this point, I'm leaning towards my 4x3 having a problem on that particular slot, and as soon as I get home from work I'm going to sit it in the bottom of the case and connect it directly into the system, bypassing the stacker.  If it works fine there, and I can get it to rebuild parity on that drive (keeping my fingers crossed), I will be ordering a higher-quality stacker..

 

If it works fine there, and I can get it to rebuild parity on that drive (keeping my fingers crossed)

 

I think you meant to say "rebulid data on that drive".

 

It may be possible that there is nothing wrong with the 500G drive that you replaced.

It actually started growing bad-sectors before it completely failed (and failed diagnostics on another machine), so it's currently winging it's way to the Seagate RMA farm (had the bad firmware on it as well).  And I definitely meant rebuild data on the new 750.  I'll most likely replace that stacker with a good Chenbro soon as I can find one in stock (and it'll give me one more drive in the same space as well, my case is completely full at the moment).

 

I just restarted the machine from work (just rebooted it, nothing else), and the drive shows up now.  Looks to me like there's definitely some sort of connectivity issue there.  I wouldn't think it was a power problem, as the 750g drive should be using less than the Seagate 500g I took out (although I REALLY should get a bigger power supply, was waiting until I got a bigger case for the system..).

 

I wasn't trying to say you were underpowered, but the wires and connectors can be marginal and create power problems. Recently someone tracked a recurring crash to a bad splitter. Connecting a drive to a different power connector helps rule out this sort of thing.

As was mentioned above, make sure you grab a syslog, especially before you reboot.  It can be very helpful in determining the real issues.

 

Also, just a recommendation, and perhaps too late:  if the drive was readable at all, I would not send it back until its replacement was up and you are sure its files are safe.  Even if a drive is failing and absolutely has to be replaced, most or all of the data can still be copied off, if something goes wrong with the replacement process.

I'm attaching a syslog from a boot where the drive isn't seen.  I still and hopeful that it's not a bad drive, but a bad tray (like I said, it didn't feel 'right' when I stuck the drive in the slot, the door didn't snap into place like the other trays).  The disk in question is listed as Disk 7, and it's connected to the stacker, then to an Adaptec 1530 controller (there are 2 other 750's on that same controller already, and a 160g I'm using as a cache drive)..

 

http://www.datai.com/private/syslog.zip

Ok, after moving the drive out of the stacker, it seems to be working now (shows up every time).  I went ahead and used the mdcmd set invalidslot 7 (got the result of cmdOper=set cmdResult=ok), and started the array.  At the moment, it's rebuilding the data on that drive, and shows that it's going to take 296 minutes (roughly the same time it said when I first put the 750 in the system).  Hoepfully everything will go fine (at least I'm keeping my fingers crossed ).

 

I am probably going to order another case today, and next week replace the existing stacker with 3 3x5 Chenbro's (not going to use a chenbro case though, I'm going to order another Antec 900 as it's currently on sale again, although I'd REALLY like the 1200, as it could be fitted with 4 3x5's, for when Unraid goes to 20 disk support .

 

 

**EDIT**  Spoke too soon, got up to about 14%, and the system died again.  System went to a black screen (I'm assuming that it's locked up with the screen-saver running), and it's unresponsive to keyboard or network.  I'm going to try pulling the 750 and sticking it in another machine just to be sure it's working correctly, although it appears to be running fine until the system's been up for a bit.  At this point I'm wondering if it might be a power problem, once all the drives are up and running (I'm using a Corsair single rail 400cx, should have gone for the 650..).

 

I cheated and propped a 2nd supply next to the machine, powered just that drive, and so far it's gone much farther than before (I'm at 20% now, before never got past 10%).  If it makes it through, I guess I'll be buying a new Corsair 650w tonight..

 

Sounds like you are definitely on the right track.  10 hard drives on even a good 400 watt PSU seems to me to be pushing it!  Drive problems won't cause a crash like that, they cause drive errors only.  Two other slight possibilities are heat and memory.  I would run a memory test just in case, and check for very hot motherboard components, particularly the CPU and bridge chipsets.  CPU temp (well, *probably* the CPU) at boot was 38C, which is pretty good, so probably not a CPU halt.  So it most likely is bad or insufficient power.

 

According to the syslog, it had no problems initially finding and configuring Disk 7, but a couple of hours later, it went down hard!  And rather surprisingly, it took less than 12 seconds for the kernel to disable the drive!  Although it continued to issue a few more exceptions, and was issued hard resets a few more times, I have never seen a disabled drive return from the dead, without a reboot.  I would generally recommend more than a reboot, powering off, to recover the drive.  You have done better by moving it to a different port.

 

I can't say absolutely that there is nothing wrong with the drive, but when they quit so fast like that, it is almost certainly not the drive's fault, so the drive is probably fine.  To be thorough, you might want to check a SMART report for the new drive, just to be sure.

Failed at about 25% (further along than normal though).  I've got the drive doing the WD extended diagnostics at the moment, and so far no errors (halfway through).  Still can't understand why the system is freezing, all I replaced was the 500 for a 750 (unless somehow the power draw difference has caused an issue with the existing power supply).  I'll keep plugging away and see if I can figure it out.  Wonder if it might be a compatibility problem with the green drives and the adaptec controller?  (The other 2 750's I have are Blues, not Greens).  Could always try it on another port I guess.

 

After finishing the WD diagnostics, no errors found (extended diagnostics).  Booting up a unraid key to check smart status, no problems there either.  I'm just plain at a loss.  I'm going to try moving it to a new port tomorrow evening and see if that helps out or not.  If it does work, I guess I'll stick one of my 640's in it's place, and the 750 in place of the 640..

 

Normally this type of problem indicates a cabling problem - either on the data side or the power side.

 

The drive diagnostics could be running a procedure local on the drive itself, that does not heavily exercise the interface.  (I'm not positive on this, but I think it likely that's how it works).

 

I have 1T green drives on an adaptec controller - incompatiblity with the controller is very unlikely.

 

If you have PSU issues, you'll normally have issues when powering up, or perhaps when spinning up all the drives in your array at once.  This is not what's happening.

 

Drives can and do fail, but if the drive has no smart attribute issues (i.e., no reallocated sectors), and comes up from a fresh boot working, it is likely NOT a failing drive.

 

I can't rule out incompatibility with your motherboard.  (Seems unlikely though).  What is your motherboard?

 

I also can't rule out controller failure.

 

As I said, I think you have a cabling problem.  If you are using backplanes / docking ports, that is part of the cabling and is therefore suspect as well.  It could also be the power adapter is bad, a power splitter is flakey, some foreign matter interfering with a contact, etc.  I'd recommend trying a different data cable (fresh if you have one) connected directly from controller to drive, and a different power port complete with different adapter/splitter.  Try to make all the connections as different from now as you can.

I noticed that in the syslog that the controller that is assigned to ATA5 & 6 seemed to be the ones giving the exceptions, so I replaced the controller with a different one I had in the drawer.  I'm currently generating parity on the 750, and it seems to be going OK for now (although, it's going slower since that other controller is a PCI sata controller, and the one I replaced was PCIE, but if it works, I'll cross that bridge when I come to it).  Keeping my fingers crossed for now..

 

*EDIT*  Since replacing that other controller (a Rosewill 2x Sata, 1x Pata PCIE controller), I've not gotten any of those exceptions at all.  It's happily generating parity at the moment (although the last few tries got to around 20% or so before the system went crazy).  The last time it did crash, the screen was actually filled with errors about the ATA5 & 6 ports having exceptions.

 

**EDIT**  Looks like that definitely fixed it, I'm up to 40% complete building parity and no errors or crashing.

 

 

Finished parity build without any errors, guess I either have a bad controller, or this one isn't supported well under linux.  I'll look at ordering another Adaptec 1430sa this week I guess..