I was running low on space, so I started the process of replacing a 250GB drive with a 1TB drive.  The system was about 10% done with the "starting" process, and at one point when I clicked refresh no webpage could be found.  I can't ping the system either, it looks like it locked up completely.

 

I did a parity check before I started to add the new drive.  I could use some advice/help on this.  I don't want to mess up all my other drives, and I'd love to not lose the data on the drive I'm upgrading.

 

My thought is to hard shutdown the system (don't think I have any other choice), put the old drive in the place of the new 1TB drive, and "Restore" instead of start.  This should make the parity match all of the drives, which should be correct since no files where added to the array since the last parity check.  Once that is complete, I can try to replace the drive again. 

 

My questions:

Is this the best way to recover?  It will certainly take a long time.

Is there anything I can look at to figure out what happened, so it doesn't happen again next time?  The only thing out of the ordinary was that I sprayed compressed air around the case to clean all the dust out of the fans.  I thought this was harmless, but it is the only thing I can think of that might not happen the next time I try the upgrade.

 

Thanks for any help,

Brett

I was running low on space, so I started the process of replacing a 250GB drive with a 1TB drive.  The system was about 10% done with the "starting" process, and at one point when I clicked refresh no webpage could be found.  I can't ping the system either, it looks like it locked up completely.

 

I did a parity check before I started to add the new drive.  I could use some advice/help on this.  I don't want to mess up all my other drives, and I'd love to not lose the data on the drive I'm upgrading.

 

My thought is to hard shutdown the system (don't think I have any other choice), put the old drive in the place of the new 1TB drive, and "Restore" instead of start.  This should make the parity match all of the drives, which should be correct since no files where added to the array since the last parity check.  Once that is complete, I can try to replace the drive again. 

 

My questions:

Is this the best way to recover?  It will certainly take a long time.

Is there anything I can look at to figure out what happened, so it doesn't happen again next time?  The only thing out of the ordinary was that I sprayed compressed air around the case to clean all the dust out of the fans.  I thought this was harmless, but it is the only thing I can think of that might not happen the next time I try the upgrade.

 

Thanks for any help,

Brett

If the server has crashed, and you cannot "ping" AND cannot get a prompt at the system console(please check... very important to know if it is responding at all) and you cannot telnet in, then your only recourse is to reboot.

 

Once rebooted, it should detect your new disk and the rebuild process should start on its own once you press the "Start" button.

 

As you said, the fallback is you still have the data on the old disk you removed from the array.

 

We con't know more until youo can get a look at the syslog.  If it is at all possible get a copy of it (why you need to verify from the system console that a login is not possible) 

If you must reboot, report on what you see on the screen before doing anything.  Odds are it will say a new disk is detected and will wait for you to press "Start" to reconstruct the old contents onto it.

 

DO NOT use the button labeled "restore" at this time.  It will invalidate parity and eliminate any chance of recovery using it.

 

If you do have to reboot, still post a syslog as a baseline.

 

Joe L.

Ok, I left everything alone and rebooted (no ping, no telnet == no choice).  After reboot, the web console shows all disks green except the new one is red.  The command area only contains "Starting..." and the Refresh button and a greyed out Start button.  I haven't hit refresh yet.  Anything I should do before hitting refresh?

 

Here's the syslog.

 

Thanks for the prompt help, it is really appreciated.

 

Brett

syslog20100114.txt

It appears as if it is starting the process of rebuilding disk3.

 

Press"refresh" on the browser window...

 

If the array is not started, check the checkbox under "Start" and press "Start"

 

Also, in a telnet window, type

tail -f /var/log/syslog

to see the tail end of the syslog and follow it as messages are added.

 

If errors occur, you'll be able to see them.

 

Joe L.

I didn't need to hit Start, when I did the refresh the "Data-Rebuild is in progress." was listed.  I've got the tail -f going in a terminal as well. 

 

I will post back (success or failure) when things change.

 

Thanks again,

Brett

Everything is still running, I had one unusual glitch where my telnet connection was lost.

 

There was an entry in the tail -f

"Jan 14 12:54:48 Tower ntpd[1336]: time reset +0.162464 s"

"Jan 14 12:55:19 Tower ntpd[1336]: synchronized to 198.186.191.229, stratum 2"

 

Then I got the command prompt with the message "Connection to host lost."

 

But I was able to reconnect, and it looks like the Data-Rebuild is still running.  When I started the tail -f again, the only new log entries were for the new telnet connection and login.

 

I know ntp is the network time protocol that synchronizes machines to a standard time.  I wouldn't expect it to cause a connection loss.  So I don't know if it means anything, but it seemed unusual enough to report....

 

Brett

Everything finished fine on the 2nd attempt.  Array is back on line, and all of the media on the replaced drive is available on the new drive.  Not that I should be surprised!  Just got scared when it froze in the middle of the update.

 

Thanks for the help Joe.

 

Brett