Hi guys, I need some help.

 

I was forced to do a hard shutdown last night. One of my drives was showing constant drive activity even though there was nothing being read from/written to it. I let it go for half an hour to see if it stopped but it didn't.

 

The disk did not unmount during normal powerdown, and consequently the server hung, and I was forced to kill the power.

 

Upon restarting the server today there were errors as expected and a data rebuild was automatically commenced.

 

To me the rebuild seems far to slow. It is fluctuating between 3000KB/s and 6000KB/s and eta's of 94-180 hours.

 

4 days minimum to rebuild data on 1 disk, Really??

 

Could the disk have gone faulty or something, that would explain the weirdness last night and the slow speeds now.

 

I do have a brand new WD20EARX I can replace it with but it has not been precleared. I have also checked the cable connections and they seem good.

 

System specs:

Unraid 4.7

3 drives: 2 x WD20EARX's, 1 x WD20EARS

Motherboard: Supermicro X7SPA-H-0

Drives are connected to onboard sata ports.

 

The offending drive is disk1 and is the only WD20EARS

 

Syslog attached.

 

.......................................

 

UPDATE

 

Alright now I've got huge problems. I ran a short smart test on the disk and saved the report to the flash drive as instructed in the wiki. When I browsed to the network share for the flash drive it was very slow opening, it took about 30 seconds from clicking the icon to actually open the flash share, then clicking other folders inside the flash took another 30 seconds to open.

 

Now the server does not show up on the network at all. The server is not visible, the webui will not load and pings to it's ip fail.

 

Is my flash corrupted??

 

WHAT DO I DO?

 

 

 

 

syslog.txt

Can you shutdown the server, pull the USB-stick and check it in another PC? Also copy the syslog then.

The server is currently shutdown. Had to kill the power again as it no longer responds to commands.

 

The flash appears fine when put in another pc, the flash is readable and everything seems to be there as normal.

 

The newest syslog on the flash is actually a little older than what I posted before, but I have uploaded it anyway, there are a few smart reports in there too which might help (there are multiple reports as it's the first time I've done one so I wasn't sure if it worked so I did it multiple times.)

syslog-20120615-170726.zip

sda shows up as invalid:

 

Jun 15 17:07:22 NAS status[3584]: D#          Model / Serial          Status        Device   

Jun 15 17:07:22 NAS status[3584]: 0  WDC WD20EARX-00 / WD-WCAZA6580478 DISK_OK        sdc     

Jun 15 17:07:22 NAS status[3584]: 1  WDC WD20EARS-00 / WD-WCAZA3646980 DISK_INVALID  sda     

Jun 15 17:07:22 NAS status[3584]: 2  WDC WD20EARX-00 / WD-WCAZAC695248 DISK_OK        sdb   

 

I guess you probably knew that?

 

Is this the parity disk? 

 

 

Maybe find some clues here:

 

http://lime-technology.com/forum/index.php?topic=16182.msg149668#msg149668

 

 

The first sign of a problem as you noted:

Jun 15 16:44:39 NAS kernel: md: recovery thread rebuilding disk1 ...

.....

Jun 15 16:45:04 NAS kernel: md: md_do_sync: got signal, exit...

Jun 15 16:45:04 NAS kernel: md: recovery thread sync completion status: -4

.....

Jun 15 17:06:58 NAS mdcmd[3426]: mdInvalidDisk=1

Jun 15 17:07:22 NAS mdcmd[3581]: diskId.1=WDC_WD20EARS-00M_WD-WCAZA3646980

Jun 15 17:07:22 NAS mdcmd[3581]: rdevNumber.1=1

Jun 15 17:07:22 NAS mdcmd[3581]: rdevStatus.1=DISK_INVALID

......

Jun 15 17:07:26 NAS rc.unRAID[3641]: umount: /mnt/disk1: not mounted

Jun 15 17:07:26 NAS rc.unRAID[3641]: Could not find /mnt/disk1 in mtab

Jun 15 17:07:26 NAS rc.unRAID[3641]: umount: /mnt/disk2: not mounted

Jun 15 17:07:26 NAS rc.unRAID[3641]: Could not find /mnt/disk2 in mtab

---RESTART

Jun 15 17:15:16 NAS kernel: md: recovery thread rebuilding disk1 ...

Jun 15 17:15:39 NAS emhttp: get_config_idx: fopen /boot/config/shares/mysql.cfg: No such file or directory - assigning defaults

Jun 15 17:15:39 NAS emhttp: shcmd (16): killall -HUP smbd

 

First, try to run check all connections, run another smart test on the EARS drive...as you haven't completed one recently, except maybe the one on the flash drive?

 

 

Jun 15 17:07:21 NAS smartctl[3552]: SMART Self-test log structure revision number 1

Jun 15 17:07:21 NAS smartctl[3552]: Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error

Jun 15 17:07:21 NAS smartctl[3552]: # 1  Extended offline    Aborted by host              90%      2623        -

Jun 15 17:07:21 NAS smartctl[3552]: # 2  Extended offline    Aborted by host              90%      2622        -

Jun 15 17:07:21 NAS smartctl[3552]: # 3  Extended offline    Aborted by host              90%      2622        -

Jun 15 17:07:21 NAS smartctl[3552]: # 4  Extended offline    Aborted by host              90%      2622        -

Jun 15 17:07:21 NAS smartctl[3552]: # 5  Short offline      Completed: read failure      10%      2622        2048627544

 

 

All connections appear fine.

 

Unfortunately I can't get any of the disks online now. If I start the server with any of the disks physically connected the server does not appear on the network at all, it can not be pinged and the webui does not work. If I physically disable the disks then start the server it is accessible via pings and through the webui, but of course it has an error that there are too many missing disks.

 

If I start the server with all disks physically disabled and then enable them when the server is up, I get this (see attachment):

 

Before this happened my disk assignments were:

 

DISK 1

WD20EARS S/N: WCAZA3646980

 

DISK 2

WD20EARX S/N: WCAZAC695248

 

PARITY

WD20EARX S/N: WCAZA6580478

 

So the disks seem to have changed their assignments and there are duplicates of each disk?

 

The sdX values are unimportant and subject to change. sda refers to the entire disk and sda1 refers to the first partition. Can you reassign the drives correctly on the unRAID devices page?

It looks like I should be able to re-assign the disks. I have the option to unassign them all in the devices page so I assume I can reassign them. But the question is should I?, I don't want to lose any data by reassigning them in the wrong sequence or something.

 

I have started preclearing my spare drive and have commenced a long smart report on the drive I suspect is faulty.

 

 

 

Here is the smart report from the EARS

 

And a current syslog is attached.

 

When viewing the syslog in unmenu there are 2 entries listed in red:

 

Jun 16 11:16:10 NAS kernel: ata1: exception Emask 0x10 SAct 0x0 SErr 0x4050000 action 0xe frozen (Errors)

Jun 16 11:16:10 NAS kernel: ata1: SError: { PHYRdyChg CommWake DevExch } (Errors)

smart1.txt

syslog.txt

It must be a hardware-error. My first guess is the harddisk is broken. But it can be connection issues, bad cables(power-data cable) or SATA-port.

Alright then. The spare disk should be done preclearing in 6 hours or so, so how do I go about getting the array back up and running?

 

Can I replace the dodgy disk with the precleared one? What order do I need to assign disks in so as not to lose an data? I basically need to get the array to trust the good data drive and the parity drive and rebuild the data onto the new drive. The wiki makes it very clear forcing the array to trust parity should only be done when all the original disks are present which will not be the case here.

 

(I'm just about to head off for work, so I'll be back in 10 hrs, and the new disk should be ready to go)

Update, so the spare disk is now precleared and ready to go.

 

I booted the server with the "bad" disk disconnected and the parity and good data disk connected. The server booted up normally and the webui shows both disks as green - array is stopped (and the "bad" disk shows as missing.......obviously). I don't know why it refused to startup properly when I did the exact same thing last night, but it seems good now.

 

So how do I proceed from here?? Do I try to rebuild data on the "bad" disk again (which will probably lead to the slow rebuild rate that initially prompted this thread) or do I replace it with the brand new precleared disk?? If I replace it exactly what procedure must I follow??

 

If it does need replacing, is it the actual disk that is faulty or is it the filesystem?? Can the disk be precleared and used again in the future (even if its just used on a windows machine or something??) or does it need to go in the rubbish bin?

Update, so the spare disk is now precleared and ready to go.

 

I booted the server with the "bad" disk disconnected and the parity and good data disk connected. The server booted up normally and the webui shows both disks as green - array is stopped (and the "bad" disk shows as missing.......obviously). I don't know why it refused to startup properly when I did the exact same thing last night, but it seems good now.

 

So how do I proceed from here?? Do I try to rebuild data on the "bad" disk again (which will probably lead to the slow rebuild rate that initially prompted this thread) or do I replace it with the brand new precleared disk?? If I replace it exactly what procedure must I follow??

 

If it does need replacing, is it the actual disk that is faulty or is it the filesystem?? Can the disk be precleared and used again in the future (even if its just used on a windows machine or something??) or does it need to go in the rubbish bin?

First step it to get back to a stable configuration.

 

If you have a pre-cleared disk, rebuild onto it.  Worry about the other disk, which might be good or bad, once stable.

 

Joe L.

Right then, I will install the new disk now and let it do a rebuild.

 

Will report back when done.

 

EDIT - Data rebuild is currently underway @ 58,000 KB/s

 

 

Data rebuild is complete and array has all greens.

 

So what's the story with this disk, is it headed for the rubbish bin??

 

Smart report is in post #8

 

Also the new disk is using the same sata and power cables as the bad disk was so we can rule out faulty cabling.

You could hook it back up and preclear it, or connect it to another pc and do a low-level format. Both would write 0 to all sectors. I wouldn't use it for important data though.

 

If its still under warranty is rma it before anything else. But I'm not a harddrive expert

 

Sent from my HTC Vivid

Now I've got a new problem. The disk that failed had a mysql db on it for xbmc.

 

All the data appears to have been rebuild onto the new disk but xbmc can not connect to the db, it simply shows a black screen. I have tried reinstalling mysql but the install fails, I assume this is because there is already a mysql folder on the disk. I am unable to delete the mysql folder from the disk, it fails with an error message to check the folder is not write protected or currently in use. Can I delete the folder through the command line somehow??

 

If I telnet in an send killall mysqld it says no process killed - so I do not think mysql is running at startup and I can not reinstall it since I can't delete the original mysql folder.

 

Sounds like a permissions issue.

You might be right, I just opened the share for that disk and tried to copy a file to it and it failed with the same error, so the whole disk is read only or something. Though I wrote a bunch of stuff to it earlier and it was fine.

 

Disk shares in unraid are set to Export read/write and I just set them to read only and then back to read/write again to see if re-enabling it would fix it but it did not.

 

There's a bunch of stuff in the syslog too that doesn't look good:

 

Currently running reiserfsck on the disk.

 

syslog.txt

Did you run the "New Permissions" script?

 

Edit: went back to your original post and saw that you're running 4.7, so this doesn't apply.

 

However, I would do a chmod and/or chown, maybe on just one folder to see if it fixes the issue.

Did some reading, apparently unraid will make a drive read only if the filesystem is corrupted.

 

The reiserfsck found errors and recommended running --rebuild-tree so that is running now.

 

 

This is just getting ridiculous now.

 

I ran reiserfsck --rebuild-tree and it completed successfully, I mounted the drive and everything seemed good. It was really early so I shut the server down and went to sleep.

 

When I went to boot the server this morning it would not boot like it normally does (the server is headless) normally the network icon in the tray on my desktop will popup a few times with link disconnected, and I know it's doing its thing normally - but this did not happen. I plugged a keyboard and monitor into the server to see what what was happening and found it to be sitting at a black screen with a flashing cursor. I rebooted it and immediately after post it hangs at the black screen with the flashing cursor.

 

I figured the flash was corrupted to I recovered it with a backup I had laying around - same issue.

 

I reformatted the flash and installed a clean Unraid 4.7 - same issue

 

I tried another flash with the backup image - same issue

 

I tried the same other flash with the clean Unraid - Same issue.

 

I have however discovered through trial and error that if the disks are physically disconnected from the server the server will boot normally.

 

But even a clean unraid install will not boot when the disks are connected - what is going on?

 

Disks connected - server hangs immediately after post

Disks disconnected - server boots normally.

 

WTF do I do now?

 

Syslog attached, though not sure what use it will be since unraid will not boot when disks are connected it can't log the problem.

syslog-20120619-140713.txt

Update, solved the booting problem.

 

It turned out the boot priority had somehow changed itself to one of the array disks, so the server was ignoring the flash drive.

 

So the server will now boot up normally. However it started with the array stopped and a red ball next to the new disk. So it is currently rebuilding the data on the new disk (again!).

 

It also looks like my flash has problems. There is quite a lag between when it is plugged in and when shows up under my computer.

 

Hopefully once that is finished everything will be good to go. Though I did have to run resierfsck after the last rebuild (I suspect that was because the EARS was unaligned whereas the EARX is 4k aligned).

 

BTW I ran WD Lifeguard on the EARS disk and it failed the long test due to bad sectors, so looks like it will have to be RMA'd

Update, solved the booting problem.

 

It turned out the boot priority had somehow changed itself to one of the array disks, so the server was ignoring the flash drive.

 

I bet you have ASUS?

 

It also looks like my flash has problems. There is quite a lag between when it is plugged in and when shows up under my computer.

 

Try connect the "head" and boot with that. You'll see what's making the lag. I have two controllers that take like forever to boot.

I bet you have ASUS?

 

No, Supermicro.

 

Anyway I have fixed everything now, everything is back up and running perfectly.

 

I did the data rebuild and the array was all green. - so one problem fixed.

 

I still had the issue with mysql not running - but I solved that too.

 

Mysql didn't work originally as (and I'm speculating here) unraid had made the entire disk read only due to the corruption caused by rebuilding a non aligned disk onto an aligned disk, and since it was read only mysql could not be installed at startup.

 

The reiserfs rebuild fixed the corruption and updated parity to match. Unfortunately since my flash died right after that was done and my last flash backup was from before the data rebuild was done, unraid didn't recognise the new disk and had to perform a data rebuild again (luckily since the reiserfs rebuild had updated parity the aligned/unaligned problem was no longer an issue.) That got the array all green again. Now the disk with the mysql db was actually writeable again I assumed mysql would work normally - but no.

 

Installing mysql manually from the unmenu package manager showed the install was failing due to not being able to locate the socket or something. I figured this was probably caused by there already being a mysql folder located on the disk and that was causing a confilct. So I backed up the mysql folder on the disk and then deleted it. Then I manually installed the mysql package which created a new mysql folder on the disk. Then I copied the contents of the mysql folder I had backed up into the new folder on the disk - overwriting all the newly created files. After a reboot mysql was up and running normally and my original db was all intact.

 

So that was a frustrating few days, and not being able to access any of the data from my server sucked - I was stuck watching whatever had downloaded to my desktop during the day. But it's all fixed now so hopefully there won't be any more problems. Thanks to everyone that offered assistance.