I've had a disk fail on me yesterday and reconstructed onto a new disk no problem. I ran a correcting parity check afterwards (today) and there were some errors, but the parity check completed successfully and array was all green. The array was then put to sleep after the conclusion of the parity check.

 

A few hours ago I woke it up and watched some tv, suddenly I lost video, so I came and took a look at the control panel and Unraid had disabled a disk due to errors (red ball), this was a different disk and not the one that was replaced earlier.

 

I had a spare disk precleared and ready to go so I replaced the failed disk and commenced a rebuild. Half way through the rebuild a different disk completely started throwing up REISERFS errors. The data rebuild on the other disk appeared to continue as normal (though I suspect it's dodgy).

 

The disk then started to throw up handle_stripe read errors and the rebuild speed on the other disk dropped to virtually zero, then the sata link on the disk with the errors started dropping out and this stopped the data rebuild.

 

Right now I have put the array to sleep to await further instruction.

 

WHAT DO I DO?

 

Should I do a reiserfs check on the disk with the errors and hope they can be repaired?

 

Should I do a resierfs check on the disk that was initially disabled (and was replaced - it was removed from the array and has not been touched since) and hope it can be repaired and then used to repair the now failing disk??

 

Just before I put the server to sleep I checked the shares and they were working and data appeared to be available, so I hope that means nothing is lost at this stage, but I am very concerned that I am on the verge of losing everything.

 

I will not do anything until I get further advice.

 

OS version 5.0 rc8a

 

 

 

 

 

 

 

 

 

syslog.zip

Well... everything is possible in the universe but the chance that you experience 3 consecutive drive failures is small, so you need to check any components underneath that are shared:

 

1) check your cabling and power connections, everything still secure? (doublecheck psu splitters for example);

2) are all the failing drives on the same SATA expansion card and/or motherboard ?

3) are all the failing drives in the same drive cage ?

4) run a memcheck (part of the booting process, its choice, attach a monitor and keyboard and you will see it

 

In the meantime keep your array spun down and do not use it. Do not start doing filesystem checks since if the error is in another component running those checks will only make for a higher chance of data loss.

 

To my eye your syslog is full with different errors pointing more towards something else that individual disks, but others are more knowledgeable then me here.

1) Cabling seems fine, everything is tight and secure in the sockets.

2) All 6 disks are on the motherboard - so same sata controller

3) The first drive (that was successfully reconstructed) is in a different cage, the one that was being rebuilt and the drive with the errors are in the same cage.

 

Try a new SATA cable.

If the data is truly irreplaceable and you have no backup, this would be the time to either invest in a drive recovery service or buy all new drives and use ddrescue or something similar to make clones of the drives as they are, then try data recovery on the new drives.

 

It's hard to tell from your posts just how valuable your data is ($$ to $$$$$) so it's hard to recommend recovery strategies that have a chance of making things worse. Has any data been written to the array during or after the first rebuild was started?

Try a new SATA cable.

I've just replaced the sata cable now, but have not woken the array back up so I don't know if that has made any difference.

 

It's hard to tell from your posts just how valuable your data is ($$ to $$$$$) so it's hard to recommend recovery strategies that have a chance of making things worse. Has any data been written to the array during or after the first rebuild was started?

 

Luckily I thought to grab all the absolutely critical files before I put the server to sleep, so critical data is safe. What remains is about 6TB of TV that I would really really prefer not to lose. (though about 90% of it is replaceable I guess). I've got an empty 2TB drive plugged into my desktop so I can attempt to grab that 10% that just can't be downloaded anymore (stuff that is obscure and a few years old) when I am advised to wake the server backup. Hopefully SMB is still working and I can grab that stuff incase it dies completely.

 

 

After the first disk to fail was reconstructed and the parity check run, the array was put to sleep. It was woken up hours later (and everything was green) a few files were then written to the array (cache drive) once they were transferred I started to watch something and an hour into that is when the second drive was red balled. Then I replaced that drive and about 2 hours into the rebuild is when the other disk started with the errors.

 

Should I pull the error drive out of the drive cage and connect the disk directly to the sata cable and power cable, incase it's the port on the drive cage that has issues?

 

 

Did upu run the memtest ?

Did upu run the memtest ?

 

No, as you said to keep the array spun down.

 

I'm waiting for someone to give me the go ahead to wake it up, and what I need to do once it is awake.

 

I have however removed the drives from the drive cages and connected each drive directly to a sata cable and power cable. It was done one drive at a time and I was very careful to ensure each drive is connected to the same sata port it was connected to before it was removed.

Memtest is an option on boot, before unraid even starts. The array will not start in this case.

Did upu run the memtest ?

 

No, as you said to keep the array spun down.

 

I'm waiting for someone to give me the go ahead to wake it up, and what I need to do once it is awake.

 

I have however removed the drives from the drive cages and connected each drive directly to a sata cable and power cable. It was done one drive at a time and I was very careful to ensure each drive is connected to the same sata port it was connected to before it was removed.

 

Compliments, I was just about to advise that, having all drives out of the cables makes sure you cut out the cages as possible issue.

 

If you are running 4.7 it is important to have the drives at the same sata port, for 5.0 that is no longer important.

 

I would now advise to start up the system and run memtest, as stated it is an option on boot and your array will not startup. Scattered errors could also be memory and it is important that you let it run.

 

Come back with the results.

 

If memtest comes back clean I would advise starting up the array without plugins (you can do that by removing them from your usb stick before boot, make a complete backup of the usb stick first to make sure you can revert should you make a mistake).

 

Then have it startup, most probably the array will not autostart and you will see a redball. Then capture syslog and bring it here. If the issue was with the cages or a piece of cabling that now has been reseated your syslog would not show any current errors. Unraid will probably still redball because it remembers the previous state.

 

So:

 

1) memtest (send results)

 

If memtest not ok: replace memory (and back to 1), another memtest)

 

If memtest ok:

 

2) back makeup from flashdrive, remove plugins from flashdrive, start unraid box

 

3) if array does not start automatically do not start it manualy, wait 10 minutes then send syslog and screenprint of main page to forum

 

Lots of luck !!

Alright, so I have woken the server up, I had to cancel the rebuild that was in progress as it was lagging the web interface. The disk that was being rebuilt is still red balled (as expected) the disk with the errors is still green, but shows 2581 errors in the web interface.

 

The SMB shares are still working so I am currently trying to grab everything that is not easily replaceable incase it red balls that second drive and I lose everything.

 

Once I have grabbed what I can I'll powerdown the array and run memtest as advised.

I ran memtest (1 pass - it took 1.5 hrs to complete) zero errors.

 

When exiting memtest the server reboots, so I was unable to remove the plugins from the flash drive, I let it boot anyway (intending to power it down once booted and remove the plugins then reboot). However unraid booted normally and now shows all drives as green (except the drive that was originally being reconstructed of course) the disk that had all the errors (disk 3) is green and no errors are listed.

 

The syslog seems error free too.

 

So what do I do now? Attempt to reconstruct disk 4 again?

syslog.zip

was the hard drive swaping done with the server in standby?

I think the errors are reset on a boot. All you can do is try a rebuild and see what happens.

 

Peter

 

was the hard drive swaping done with the server in standby?

Yes, server was asleep when drives were removed from the cages - however I was very careful to keep each disk plugged into the same sata port.

 

I think the errors are reset on a boot. All you can do is try a rebuild and see what happens.

 

I'll power it down for tonight and I'll attempt a rebuild tomorrow, when I can keep an eye on it.

succes !  the new SATA might have helped..

succes !  the new SATA might have helped..

 

Wrong thread?

No... but if the syslog is not showing anymore errors there is a chance that it was fixed, one ofthe things that was done was exchanging a cable. I was wishing succes.

this could be your problem having it on standby and changing a bad disk and unpluging and replugging

computer should be OFF no sleep/standby unless its server grade hardware

 

 

was the hard drive swaping done with the server in standby?

Yes, server was asleep when drives were removed from the cages - however I was very careful to keep each disk plugged into the same sata port.

 

One thing I didn't notice before. Some of your drives are around 40C with the server just sitting idle. I've read the WD Green drives tend to start having read problems with temps > 50C. So, watch the temps when you start rebuilding.

 

 

No... but if the syslog is not showing anymore errors there is a chance that it was fixed, one ofthe things that was done was exchanging a cable. I was wishing succes.

 

Oh, OK. Ya it might be success, but it can't be confirmed until a rebuild is attempted and heavy disk activity is started.

One thing to consider, unRAID uses ReiserFS which is a standard Linux file system. It might be worth running up a Linux box, maybe even from a liveCD, and mounting these drives one by one as ReadOnly and attempting to copy off the files that are most critical. Using a standard fs is one of the reasons why unRAID is a good choice, you can more easily recover data from individual drives....

Well everything was looking good, rebuild was underway.

 

Now disk 3 is throwing up errors again. The rebuild is still continuing at around 100MBs so I'm letting it run for now. But will it actually rebuild correctly when one of the disks has errors?

 

I'm not really sure what I can do to bring the drive temps down, the drives are still sitting in their trays but have been removed from the cages and are sitting elevated on some wood to allow for airflow underneath. Perhaps remove them from the trays and see if I can find a desk fan or something. It's just unfortunate its a 30 degree day today and I don't have aircon.

 

syslog.zip

Not long after my last post, drive 3 started spitting out more errors, I could hear the disk seeking constantly and the data rebuild dropped to a crawl, so I have cancelled the rebuild.

 

Right now I am backing up all the data off disk3 and as much from disk 4 as I can (disk 4 is only 100 or so movies and I can live without them).

 

Also I was looking through my records and disk 3 was actually a replacement disk from WD sent to replace another WD20EARS I RMA'd because it failed too.

 

 

Get smart reports (smart attributes) for both drives.. We should see there if there is physical hardware error.

Smart test on disk 3 would not complete (stayed at 90% for 10 minutes - and I aborted). Simple features displayed smart data for the drive so I grabbed the report anyway

 

Disk 4 completed no problems

 

Reports attached

................................................................

 

The report on disk 3 shows 44 pending sectors, so it looks like the disk has major problems right??

 

If that is the case then am I right in thinking I'll have to unassign and remove disk 3 completely. This will mean I'll be unable to reconstruct disk 4 so that data will be gone too?

 

So once backing up is complete (I should get all of disk 4 and 80% of disk 3 - the other 20% can be replaced easily), should I remove disk 3 and 4 completely, recontruct parity with just disk 1 and 2 present, then re-add disk 4 and copy the backed up data on to it, then purchase another disk to replace disk 3 and copy backed up data to it?

 

smart3.txt

smart4.txt