Started a rebuild a few hours ago. Web GUI is now inacessible.


Recommended Posts

Long story short one of my disks failed recently (bad sectors). I RMA'd it, received a replacement and precleared it.

 

I started a rebuild using the replacement drive this morning and after returning from a few errands I cannot access the web gui.

 

Shares and docker containers are working.

 

Is it possible to check on the progress of the rebuild via the commandline?

 

Is there any way to collect diagnostics without the web gui?

 

I have copied the file /var/log/syslog off in case that will yield anything helpful.

syslog.txt

Link to comment

What version of UnRAID are you using?  [Hopefully you remember, since you can't access the GUI  :) ]

 

... and is the rebuild still running okay? => i.e. look at your disks and see if all of their indicator lights are active ... or, if you don't have individual lights for the disks, see if the disk activity light on the system on.

 

Link to comment

UNRAID 6.1.9.

 

Only 1 HDD light on the front of my case. It is lit and not flickering. I'm not sure what that means, but I think it means constant disk activity.

 

It's my understanding that I can't restart the webGUI without rebooting the server or at the very least stopping the array. I would rather not do that if the rebuild is progressing.

Link to comment

mdcmd status | egrep "mdResync"

 

 

It will display some values, including:

 

mdResync=total parity size

mdResyncPos=current position

 

Thanks.

 

mdResync=7814026532

mdResyncPos=659798164

 

I will keep an eye on it.

 

If it doesn't appear to be progressing, can I do a clean powerdown (via the powerdown plugin) and attempt to restart the rebuild after reboot without worrying about losing data?

Link to comment

As long as the activity light is soliid, it's likely that the rebuild is progressing just fine, and the GUI is hung.

 

Sometimes if you reboot your PC (NOT the server), or try to access it from another PC, it will respond.

 

But basically I would just wait long enough for the rebuild to finish, and then reboot the server via a quick push of the power button (Do NOT hold it in -- a single push should cause a graceful shutdown).

 

Is there a monitor attached to the server?  ... if so, what does it show?  ... If not, do NOT "hot plug" a display while it's on -- just leave it as is.

 

The rebuild can take MANY hours -- depends on the size of the disk you're rebuilding and what the platter densities of your various disks are.    The rate can be anywhere from 3 to 6 hours per TB.    So figure how long that would be for the disk you're rebuilding, and just do nothing until that much time has passed.  [e.g. if it's a 3TB drive, wait for 18 hours and see if the disk light is still solid]    Easy to tell when it's done => the disk activity light will go out  :)

 

By the way, as I assume you know, it's best to not use the array otherwise during this process -- that will just slow things down.

 

 

Link to comment

Clearly you have both an attached display and a keyboard  :)

 

The command Johnnie gave you lets you monitor the rebuild ... so it'll be easy to tell if it hangs.

 

As long as it's progressing, I'd just let it run.

 

If you DO end up having to force a shutdown, ASK first ... as it's very important that you don't end up in a position that causes a parity sync to run on reboot => THAT would kill the ability to redo the rebuild.    Basically, if powerdown does a "clean" shutdown, you should be okay; but if you end up having to force a shutdown (i.e. by holding the power button), then you have to do it a different way to still be able to rebuild your disk.

 

Link to comment

Web GUI is back up. A bit slow to respond, but I'm content to wait for now. I'm familiar with rebuild rates. I've had disks fail before, and this is an 8TB drive. So it will be a couple of days most likely.

 

There is a monitor attached. It just shows a login prompt.  No error messages since I booted with the new drive installed this morning.

 

I also have SSH access.

 

Thanks to everyone who responded. I'm going to just leave things as they are for now in hopes that everything completes successfully.

 

Link to comment

I ran into the same behavior today with 6.1.9.  I started a parity rebuild a removing a drive.

 

When the main page reloaded, the icons for parity (orange triangle) and circles for the data disks were not displayed.  When I attempted to refresh the page, it wouldn't load.  However, I also could not connect using SSH.  I was able to log in using a keyboard and the rebuild was still in process and the array was mounted fine (/mnt/user was populated).  After a few minutes, both the GUI and SSH became responsive.

Link to comment

The rebuild made it past 4 TB last night. At this point there are only 2 drives left in the rebuild process. My 8TB parity drive, and the 8TB data drive being rebuilt.  Since all my other drives, which are 4TB, are no longer contributing to parity shouldn't this effectively be a matter of copying mirrored data off the parity drive onto the new drive?

 

The reason I ask is that the rebuild is going much slower than I would expect. Only a few MB/s. I would expect it to be able to maintain speeds over 100 MB/s at this point.

Link to comment

UnRAID actually still computes the parity values ... even if there is only ONE drive remaining (i.e. a parity drive) in the calculation.    But Yes, I'd expect it to be maintaining speeds well over 100MB/s with only a pair of 8TB drives left in the calculation, unless one of them is connected to a very slow controller (e.g. an old PCI controller) -- but that seems very unlikely.

 

I'd let it complete, however, since it's still moving along.

 

One other thought:  Do you have other activity on the array during this rebuild?  That will very appreciably slow things down.

 

Link to comment

I had to resist the urge to reply back "Of course I'm not using the array" and go back and double check to be sure.

 

Well its a good thing I did. I had left my MusicBrainz docker container running.  And of course, it was using disk7 for its /data location. Disk7 is the very disk being rebuilt and with the constant reads and writes that musicbrainz does to its DB, it wasn't doing the rebuild any favors.

 

Now that the container is stopped the rebuild is easily sustaining speeds >100MB/s, even topping 150 MB/s occasionally.

 

Its always something you don't bother checking. Reminds me of the sysadmin haiku:

It's not DNS.

There's no way it's DNS.

It was DNS.

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.