Hello Everyone,

 

I am having problems rebuilding a failed drive:

 

Everything was running fine but my array was nearly full (down to 400gigs left) I got desperate to add more space and had a spare laptop drive (250 gigs) laying around and popped it into my raid array, it past pre-clear, and so I loaded up some data onto the drive about 150gigs worth. Well Everything seemed to work and then that drive red-balled. I ran a smart test, and although the drive had no relocated sectors or pending sectors, it did have a bunch of smart errors. I googled them and others said there would be eminent drive failure and to replace the drive. I gathered together the 150$ and bought a new 3tb drive, and thought all my problems were done. They were not.

 

I pre-cleared the new 3tb drive first (it passed with no issues) and then powered down the array, and swapped the laptop drive with the new one. When Unraid came back up, it realized the change, and I choose the option to rebuild and enlarge the drive, I did this and it failed, I figured this was just a fluke, rebooted and tried again, and it failed again. I then thought that it maybe an issue with the slot the drive was in (I have a nortek 4224 using mini SAS using an AOC-SASLP-MV8) and shutdown and changed slots and then restarted and attempted to rebuild parity again, the drive got a lot farther (was failing around 10%) now it was failing around 70%.

 

The syslog gives the following errors:

Dec  2 03:02:40 UNRAID kernel: handle_stripe write error: 761660384/12, count: 1

Dec  2 03:02:40 UNRAID kernel: md: disk12 write error

 

I have learned through the forums to do smart checks on drives when I have problems and when I do so after a failure i get the following back in the smart report.

smartctl 5.40 2010-10-16 r3189 [i486-slackware-linux-gnu] (local build)

Copyright © 2002-10 by Bruce Allen, http://smartmontools.sourceforge.net

 

Smartctl: Device Read Identity Failed (not an ATA/ATAPI device)

 

A mandatory SMART command failed: exiting. To continue, add one or more '-T permissive' options.

 

However If I reboot and run a smart report it comes out great. I have attached my syslog, and smart reports for the drive, any help would be greatly appreciated!

 

Thank You for reading this.

syslog-2012-12-02.txt

smartsdj.txt

smartsdj-error.txt

Could be a cable or power issue. Which PSU? Can you put the drive in a slot that used a different SAS connection?

Hello Dgaschk,

 

As always thank you for your help, you have saved my arse numerous times already :-)

 

I am currently using the 600 watt OCZ600MXSP

http://www.ocztechnology.com/ocz-500w-700w-modxstream-pro-power-supply.html

 

I cannot at this time try another cable as all others are in use, and I have one drive that is using that backplane.

 

I am currently powering 14 drives of miscellaneous power requirements, so it is possible that I may be reaching the upper limit of my PSU.

 

I of course want to get this fixed and open to purchasing new hardware if necessary, please let me know if you think my PSU is adequate.

 

Thank You

The PSU is not sufficient for the number of drives in the system. Only 18Amps are available to power the drives. A PSU with a single 12V rail is preferred. The amperage on the 12V rail is more important than than total wattage. See here: http://lime-technology.com/forum/index.php?topic=12219.0

Hello Dgaschk,

 

I just checked and I have the 600w version which has 25A per rail. Do you think that would make a difference.

http://www.ocztechnology.com/images/awards/mxsp_wattage_charts.jpg

 

Thanks for all your advice, you have no idea how much I appreciate it!

No. You need at least 35A for the current drives. Most, possibly all, of your drives are low power. Get more amperage if you plan on adding any more drives. See the PSU thread I linked.

I want to future proof my setup so that I can max out 24 drives, and I am completely ok with buying an overpowerd PSU to make sure I don't run into any more of these problems. With that said I think I am going to purchase this power supply:

 

SeaSonic X-1250 1250W

http://www.newegg.com/Product/Product.aspx?Item=N82E16817151109

 

Here are its ratings:

+3.3V@25A, +5V@25A, +12V@104A, [email protected], +5VSB@3A

 

Ill update this thread when I replace the PSU.

 

Thanks!

Hello dgaschk,

 

Just a quick update:

I replaced the PSU with the beast I mentioned in my last post SeaSonic X-1250 1250W. I tried a rebuild and got the same problem, basically I start the rebuild go to sleep and when I come back Unraid is completely locked up. Can't access the web interface/terminal/and the screen is just black on the console.

 

The good news is we now know it's not a power issue, I can also buy a new reverse breakout sas cable, but I already have a drive on that, which runs fine, if you think that is the issue I can replace that as well.

 

What would you suggest our next move be?

 

Thank You

If the server is 'locking up' during a lengthy rebuild, please try again, but set up first to capture the system log. 

There are two ways to do this:

 

1) via webGui: click on 'Log' in the menu bar.  This will open a separate window that will display system log entries as they occur.

 

2) via telnet: open a telnet session to your server and then type this command:

 

tail -f /var/log/syslog

 

This will display system log entries as they occur in the telnet window.

 

Of the two probably 2) is preferred because it might capture more information with a true crash since it's a "shorter path" of code to execute in the telnet window than sending to your browser (though not by much).

 

When the server locks up, select/copy/paste the captured system log and post here.

Hello Limetech, dgaschk,

 

I did as Limetech recommended and tried again while having the terminal continuously recording the syslog, and have attached what I could copy from the terminal to this post. But to summarize I am basically getting the error:

 

Dec  6 16:55:34 UNRAID kernel: handle_stripe read error: 231707912/8, count: 1
Dec  6 16:55:34 UNRAID kernel: md: disk8 read error
Dec  6 16:55:34 UNRAID kernel: handle_stripe read error: 231707920/8, count: 1
Dec  6 16:55:34 UNRAID kernel: sd 3:0:4:0: [sdn] Unhandled error code
Dec  6 16:55:34 UNRAID kernel: sd 3:0:4:0: [sdn]  Result: hostbyte=0x04 driverbyte=0x00
Dec  6 16:55:34 UNRAID kernel: sd 3:0:4:0: [sdn] CDB: cdb[0]=0x28: 28 00 0d cf 9e 80 00 02 00 00
Dec  6 16:55:34 UNRAID kernel: end_request: I/O error, dev sdn, sector 231710336
Dec  6 16:55:34 UNRAID kernel: sd 3:0:4:0: [sdn] Unhandled error code
Dec  6 16:55:34 UNRAID kernel: sd 3:0:4:0: [sdn]  Result: hostbyte=0x04 driverbyte=0x00
Dec  6 16:55:34 UNRAID kernel: sd 3:0:4:0: [sdn] CDB: cdb[0]=0x28: 28 00 0d cf a0 80 00 00 d8 00
Dec  6 16:55:34 UNRAID kernel: end_request: I/O error, dev sdn, sector 231710848

 

So now I am super concerned because if this disk fails, im pretty much guaranteed to loose data right?

 

I ran a smart report on the drive and have attached it, but the funny thing is it looks totally clean.

 

I just want to thank you guys again for the awesome support! And can't wait for your next advice.

 

Thanks

Syslog.txt

smartsdn.txt

Is disk8 on the same cable as disk12?

Hello dgaschk,

 

Disk8 and disk12 are on separate SAS cables, but are using the same SUPERMICRO AOC-SASLP-MV8 controller.

 

Thank You

Do an overnight memtest.

Hello Limetech, dgaschk,

 

I did as Limetech recommended and tried again while having the terminal continuously recording the syslog, and have attached what I could copy from the terminal to this post. But to summarize I am basically getting the error:

 

Dec  6 16:55:34 UNRAID kernel: handle_stripe read error: 231707912/8, count: 1
Dec  6 16:55:34 UNRAID kernel: md: disk8 read error
Dec  6 16:55:34 UNRAID kernel: handle_stripe read error: 231707920/8, count: 1
Dec  6 16:55:34 UNRAID kernel: sd 3:0:4:0: [sdn] Unhandled error code
Dec  6 16:55:34 UNRAID kernel: sd 3:0:4:0: [sdn]  Result: hostbyte=0x04 driverbyte=0x00
Dec  6 16:55:34 UNRAID kernel: sd 3:0:4:0: [sdn] CDB: cdb[0]=0x28: 28 00 0d cf 9e 80 00 02 00 00
Dec  6 16:55:34 UNRAID kernel: end_request: I/O error, dev sdn, sector 231710336
Dec  6 16:55:34 UNRAID kernel: sd 3:0:4:0: [sdn] Unhandled error code
Dec  6 16:55:34 UNRAID kernel: sd 3:0:4:0: [sdn]  Result: hostbyte=0x04 driverbyte=0x00
Dec  6 16:55:34 UNRAID kernel: sd 3:0:4:0: [sdn] CDB: cdb[0]=0x28: 28 00 0d cf a0 80 00 00 d8 00
Dec  6 16:55:34 UNRAID kernel: end_request: I/O error, dev sdn, sector 231710848

 

So now I am super concerned because if this disk fails, im pretty much guaranteed to loose data right?

 

I ran a smart report on the drive and have attached it, but the funny thing is it looks totally clean.

 

I just want to thank you guys again for the awesome support! And can't wait for your next advice.

 

Thanks

 

A bit confused...

In your first post you show disk12 with a write error (as does the attached syslog).

In this post you show disk8 getting read errors.

 

Please clarify, what is your current configuration and which disk is the new disk that you're trying to rebuild.

Hello Limetech,

 

You are correct, it appears as though both drives are getting write errors.

 

The smart test for disk 12 showed the following:

Reallocated_Sector_Ct = 0

Current_Pending_Sector = 0

UDMA_CRC_Error_Count = 0

Had some logged smart errors.

 

This was the original drive I had issues with and was the drive i was attempting to replace.

 

Upon rebuild unraid keeps locking up, I did as you suggested and ran a real time log and found that disk 8 is also apparently getting write errors, this was unknown to me until i followed your instructions.

 

The smart report for disk 8 showed the following:

Reallocated_Sector_Ct = 0

Current_Pending_Sector = 0

UDMA_CRC_Error_Count = 0

No logged smart errors.

 

So the question is, if the drive is passing smart with flying colors, why is it getting write errors?

 

My server hardware stats are in my signature and I have attached a picture of my configuration page.

 

Please let me know if you need anything else to help me, and I would be glad to provide it.

 

Thanks

 

 

 

 

 

 

Hello Limetech,

 

You are correct, it appears as though both drives are getting write errors.

 

The smart test for disk 12 showed the following:

Reallocated_Sector_Ct = 0

Current_Pending_Sector = 0

UDMA_CRC_Error_Count = 0

Had some logged smart errors.

 

This was the original drive I had issues with and was the drive i was attempting to replace.

 

Upon rebuild unraid keeps locking up, I did as you suggested and ran a real time log and found that disk 8 is also apparently getting write errors, this was unknown to me until i followed your instructions.

 

The smart report for disk 8 showed the following:

Reallocated_Sector_Ct = 0

Current_Pending_Sector = 0

UDMA_CRC_Error_Count = 0

No logged smart errors.

 

So the question is, if the drive is passing smart with flying colors, why is it getting write errors?

 

My server hardware stats are in my signature and I have attached a picture of my configuration page.

 

Please let me know if you need anything else to help me, and I would be glad to provide it.

 

Thanks

 

Please post screen shot of 'stock' webGui Main page.

 

SMART is notoriously unreliable.  Most drives fail without SMART reporting any issues at all.

 

Good read:

http://static.googleusercontent.com/external_content/untrusted_dlcp/research.google.com/en/us/archive/disk_failures.pdf

 

Good summary of above good read:

http://www.dba-oracle.com/t_disk_average_failure_rate_afr.htm

 

Hello Limetech,

 

I am running simple features, so i'm not exactly sure it is stock. I am unsure on how to remove simple features. I tried to look it up but could not find any info on how to boot to the default interface instead. I have Attached a screen shot of the simple features configuration page, if that is not adequate, please let me know, and I can try and dig further and figure out how to disable it.

 

Just an FYI, I do have other plugins such as couchpotato, marachino, headphones, sabnzbd, sickbeard, darkstat, and airvideo.

However I have disabled them all so they are not running, as I was afraid they may have been causing the issue.

 

Thanks

Hey Limetech, dgaschk,

 

So I think at this point, I am most likely not going to be able to rebuild the array. (Unless you know something I don't) So I imagine that the fix is going to be pulling the bad drives from the array, and rebuilding parity. Then I can attempt to recover data from the individual file systems of the corrupt drives and add that data back to the protected array.

 

Can you please assist me by providing the best way to achieve that? I did find an article:

 

http://lime-technology.com/wiki/index.php/FAQ_remove_drive

 

That explains how to remove a drive, but I want to make sure that it is up to date, before I begin, as I don't want to cause further problems for myself.

 

Thanks guys!

There is a New Config utility in version 5. Look in the Utilities tab. Record the drive serial numbers then click new config.

 

First try to read the disks in another system. Install a reiserfs driver and try to read the disks. There a number of Windows recovery utilities for reiserfs.

Update:

Since Disk 8 was still mounted and part of the array, I was able to successfully copy all data from it to an external HDD.

 

I did not have the same luck for disk 12, I will need to pop that into my windows machine and install a reiserfs driver to see about extracting the data.

 

I took a screenshot of the drive configuration, then pulled disk 8 and disk 12. Then I used the new config utility to wipe the array configuration.

 

I then went back through and put in all the drives in the correct order and put a new 3 tb drive in the slot where disk 8 was.

 

The array is now started, and Im rebuilding parity, and I can access all my data.

 

I will provide another update when parity is rebuilt, and will mark this thread as 'solved'

 

Thank You for all your help Dgaschk and Limewire!

Hello Limewire, Dgaschk,

 

Bad News

Unraid is failing on trying to rebuild parity. Now it says that I am getting read errors on the brand new drive.

 

1. I ran an all night Memtest which ran for 19 hours, it found no errors so I think my Ram is good, You can see the test here:

http://i.imgur.com/sCBX7.jpg

 

2. I have attached the smart report for the new drive to this post.

 

3. I have attached the log file I was able to grab, but there are parts missing, When it fails parity it completely locks up as it was doing before when attempting to rebuild a drive. So I had to use the command: tail -f /var/log/syslog

I grabbed everything I could.

 

My Thoughts on what to try next:

1. Pull the new drive since it has no info anyways, and then wipe the config and rebuild parity again, then add the new drive to the array.

2. Downgrade to beta14 to see if I can rebuild parity using that version.

 

Please let me know what you think the next best option is.

 

Thanks Guys

smartsdm.txt

Syslog.txt

Update:

I pulled the new drive and am rebuilding parity again, this time its been going for 4 hours so I think its going to complete without the new drive. At least Ill have parity again! Woohoo!

 

However there are still some concerns, for one I get this every so often in the log and I have no idea what it means:

 

Dec 13 14:48:21 UNRAID kernel: sas: sas_eh_handle_sas_errors: task 0xf6a3ca00 is aborted (Errors)

Dec 13 14:48:21 UNRAID kernel: sas: ata13: end_device-0:2: cmd error handler (Errors)

Dec 13 14:48:21 UNRAID kernel: sas: ata11: end_device-0:0: dev error handler (Errors)

Dec 13 14:48:21 UNRAID kernel: sas: ata12: end_device-0:1: dev error handler (Errors)

Dec 13 14:48:21 UNRAID kernel: sas: ata13: end_device-0:2: dev error handler (Errors)

Dec 13 14:48:21 UNRAID kernel: ata13.00: exception Emask 0x0 SAct 0x4 SErr 0x0 action 0x6 frozen (Errors)

Dec 13 14:48:21 UNRAID kernel: ata13.00: failed command: READ FPDMA QUEUED (Minor Issues)

 

And I am also concerned that if I was getting read errors on the new drive should I attempt to pre-clear it again and add it to the array, is my new drive bad? Why would it rebuild parity without it, but not when it is in?

 

Any help is greatly appreciated.

 

Thanks

 

 

Hello Everyone,

 

After about 6 hours of rebuilding parity, it failed stating I have write errors on disk 9, which is another newer 3 TB drive. It is highly unlikely that I have this many failed drives. Something else has to be going on, either its a cable, my AOC controller, or there is something wrong with Unraid 5 RC8.

 

Attached is the log I was able to pull stating that there were write errors on disk 9.

 

The only things to try that I have left are the following;

 

1. Replace cable

2. Downgrade to Beta 14.

3. Replace SUPERMICRO AOC-SASLP-MV8

 

As always any suggestions would be greatly appreciated.

 

Thank You

Syslog2.txt

Hello Limewire, dgaschk,

 

I broke down and reverted back to RC5-Beta14, from a backup I had taken.

I was able to rebuild parity using the exact same hardware setup that failed on 5.0-RC8a.

 

When using beta 14, I was one of the people that kept suffering from the "BLK_EH_NOT_HANDLED" problem.

here are the logs to my beta14 parity rebuild, as they were too big to attach: http://pastebin.com/gBJqPy9k

 

Here is the Forum Post outlining the "BLK_EH_NOT_HANDLED" problem:

http://lime-technology.com/forum/index.php?topic=20529.0

 

And while using beta 14 the problem would come up in the log very often.

 

I had a lot installed on my Unraid 5.0-RC8a, such as:

 

1. Simple Features

2. Darkstat

3. Couchpotato

4. Marachino

5. Sickbeard

6. Sabnzbd

7. OpenSSL

8. Air Video

 

However I did my best to shutdown all of those programs from running, during parity restore.

Do you think any of those programs could have interfered with the rebuild? It doesn't seem like they would to me, but then what do I know about the inner workings of Unraid? Answer: Not much.

 

So my main questions are:

 

1. Should I just get rid of my SUPERMICRO AOC-SASLP-MV8? Because it does not appear that they are very compatible with Unraid.

 

2. I have a forum post open in the hardware section, thinking about replacing it with a LSI00244

http://lime-technology.com/forum/index.php?topic=24733.0

 

Is that a good idea, would Unraid be able to utilize the card?

 

Thanks