Sooo after several years of hiccup free uptime, i've got my first real major [potential] drive failure.

 

I am running 5.0-rc16c and using SimpleFeatures. This is on an ESXi setup - but at least from what I can tell everything else is fine.

 

I got a dreaded Array fault email. Syslog attached; below is what was in the email.

 

Looked at wiki and executed

smartctl -a -A /dev/sdc | todos >/boot/smart.txt

.

 

Returns:

Smartctl open device: /dev/sdc failed: No such device

 

works fine on all the other drives.

 

Checked cables/power. Everything OK.

 

One thing that's weird is the UI shows LOTS of reads and LOTS of writes to just this one drive (image attached). Also, this drive is almost 100% full. I'm HOPING that something tried to write to the drive, but since it's full, threw an error. Though I thought UnRaid handles this gracefully.

 

What do? Run a parity sync?

 

Output of /proc/mdcmd:

-----------------------------------------------------------------

sbName=/boot/config/super.dat

sbVersion=2.1.6

sbCreated=1367791154

sbUpdated=1386795234

sbEvents=150

sbState=0

sbNumDisks=8

sbSynced=1386588687

sbSyncErrs=0

mdVersion=2.1.6

mdState=STARTED

mdNumProtected=8

mdNumDisabled=1

mdDisabledDisk=1

mdNumInvalid=1

mdInvalidDisk=1

mdNumMissing=0

mdMissingDisk=0

mdNumNew=0

mdResync=0

mdResyncCorr=1

mdResyncPos=0

mdResyncDt=0

mdResyncDb=0

diskNumber.0=0

diskName.0=

diskSize.0=1953514552

diskState.0=7

diskId.0=TOSHIBA_DT01ACA200_13E0B47VS

rdevNumber.0=0

rdevStatus.0=DISK_OK

rdevName.0=sdb

rdevSize.0=1953514552

rdevId.0=TOSHIBA_DT01ACA200_13E0B47VS

rdevNumErrors.0=0

rdevLastIO.0=1386798172

rdevSpinupGroup.0=0

diskNumber.1=1

diskName.1=md1

diskSize.1=1953514552

diskState.1=4

diskId.1=WDC_WD20EARX-00PASB0_WD-WCAZAJ087969

rdevNumber.1=1

rdevStatus.1=DISK_DSBL

rdevName.1=sdc

rdevSize.1=1953514552

rdevId.1=WDC_WD20EARX-00PASB0_WD-WCAZAJ087969

rdevNumErrors.1=2

rdevLastIO.1=1386795234

rdevSpinupGroup.1=0

diskNumber.2=2

diskName.2=md2

diskSize.2=1953514552

diskState.2=7

diskId.2=WDC_WD20EARX-00PASB0_WD-WCAZAJ142017

rdevNumber.2=2

rdevStatus.2=DISK_OK

rdevName.2=sdd

rdevSize.2=1953514552

rdevId.2=WDC_WD20EARX-00PASB0_WD-WCAZAJ142017

rdevNumErrors.2=0

rdevLastIO.2=0

rdevSpinupGroup.2=0

diskNumber.3=3

diskName.3=md3

diskSize.3=976762552

diskState.3=7

diskId.3=SAMSUNG_HD103SJ_S2AEJ1BZ600843

rdevNumber.3=3

rdevStatus.3=DISK_OK

rdevName.3=sde

rdevSize.3=976762552

rdevId.3=SAMSUNG_HD103SJ_S2AEJ1BZ600843

rdevNumErrors.3=0

rdevLastIO.3=1386798172

rdevSpinupGroup.3=0

diskNumber.4=4

diskName.4=md4

diskSize.4=1953514552

diskState.4=7

diskId.4=WDC_WD20EARS-00MVWB0_WD-WCAZA5947524

rdevNumber.4=4

rdevStatus.4=DISK_OK

rdevName.4=sdf

rdevSize.4=1953514552

rdevId.4=WDC_WD20EARS-00MVWB0_WD-WCAZA5947524

rdevNumErrors.4=0

rdevLastIO.4=1386798167

rdevSpinupGroup.4=0

diskNumber.5=5

diskName.5=md5

diskSize.5=1953514552

diskState.5=7

diskId.5=WDC_WD20EARS-00MVWB0_WD-WCAZA5944835

rdevNumber.5=5

rdevStatus.5=DISK_OK

rdevName.5=sdg

rdevSize.5=1953514552

rdevId.5=WDC_WD20EARS-00MVWB0_WD-WCAZA5944835

rdevNumErrors.5=0

rdevLastIO.5=0

rdevSpinupGroup.5=0

diskNumber.6=6

diskName.6=md6

diskSize.6=1953514552

diskState.6=7

diskId.6=WDC_WD20EARS-00MVWB0_WD-WCAZA5949729

rdevNumber.6=6

rdevStatus.6=DISK_OK

rdevName.6=sdh

rdevSize.6=1953514552

rdevId.6=WDC_WD20EARS-00MVWB0_WD-WCAZA5949729

rdevNumErrors.6=0

rdevLastIO.6=0

rdevSpinupGroup.6=0

diskNumber.7=7

diskName.7=md7

diskSize.7=976762552

diskState.7=7

diskId.7=ST31000524AS_6VPBQN6K

rdevNumber.7=7

rdevStatus.7=DISK_OK

rdevName.7=sdi

rdevSize.7=976762552

rdevId.7=ST31000524AS_6VPBQN6K

rdevNumErrors.7=0

rdevLastIO.7=0

rdevSpinupGroup.7=0

One thing that's weird is the UI shows LOTS of reads and LOTS of writes to just this one drive (image attached). Also, this drive is almost 100% full. I'm HOPING that something tried to write to the drive, but since it's full, threw an error. Though I thought UnRaid handles this gracefully.

 

What do? Run a parity sync?

Put in a new drive, and let unraid rebuild onto the new drive from the rest of the drives. Unraid is handling it gracefully, it's allowing you to run with one drive failed, and will rebuild the contents on the new drive when you put it in.

 

It's possible that the drive is still ok, could be power or sata connection, but as of right now, the contents of that physical drive isn't relevant, as unraid has been emulating it using parity ever since a write to it failed.

 

You can try rebuilding it onto itself, but if I were you, I'd put in a new tested drive, and run diagnostics on the old drive after the array is healthy again.

One thing that's weird is the UI shows LOTS of reads and LOTS of writes to just this one drive (image attached). Also, this drive is almost 100% full. I'm HOPING that something tried to write to the drive, but since it's full, threw an error. Though I thought UnRaid handles this gracefully.

 

What do? Run a parity sync?

Put in a new drive, and let unraid rebuild onto the new drive from the rest of the drives. Unraid is handling it gracefully, it's allowing you to run with one drive failed, and will rebuild the contents on the new drive when you put it in.

 

It's possible that the drive is still ok, could be power or sata connection, but as of right now, the contents of that physical drive isn't relevant, as unraid has been emulating it using parity ever since a write to it failed.

 

 

 

You can try rebuilding it onto itself, but if I were you, I'd put in a new tested drive, and run diagnostics on the old drive after the array is healthy again.

 

Thanks for the reply!

 

okay - i have a drive that's PRETTY empty. how funny that for about a year i ran with it in place, not being used/assigned. about a week or two ago, i assigned it to the array. I can take the filesthat are on that, move it to another disk on the array, and do this. i'm safe to move within the array in this condition right?

 

This is me not panicking :-) - I have faith in UnRaid!

 

EDIT: Disk 6 is completely empty. so I'll shrink the array using [http://lime-technology.com/wiki/index.php/Shrink_array].

 

tonight was supposed to be about replacing my HBA and adding in: 4TB, and two 3TB drives. Good thing this went south before i even STARTED any of that.

I deleted the syslog attachment... this seems to be the pertinent place where it failed:

 

Dec 11 08:24:23 Rigel kernel: mdcmd (180): spindown 2

Dec 11 08:24:33 Rigel kernel: mdcmd (181): spindown 1

Dec 11 08:25:04 Rigel kernel: mdcmd (182): spindown 5

Dec 11 12:30:45 Rigel kernel: mdcmd (183): spindown 2

Dec 11 12:30:46 Rigel kernel: mdcmd (184): spindown 5

Dec 11 13:40:36 Rigel kernel: mdcmd (185): spindown 1

Dec 11 13:41:07 Rigel kernel: mptscsih: ioc0: attempting task abort! (sc=f769f6c0)

Dec 11 13:41:07 Rigel kernel: sd 1:0:1:0: [sdc] CDB:

Dec 11 13:41:07 Rigel kernel: cdb[0]=0x85: 85 06 20 00 00 00 00 00 00 00 00 00 00 40 e0 00

Dec 11 13:41:11 Rigel kernel: mptbase: ioc0: LogInfo(0x31140000): Originator={PL}, Code={IO Executed}, SubCode(0x0000) cb_idx mptscsih_io_done

Dec 11 13:41:11 Rigel kernel: mptscsih: ioc0: task abort: SUCCESS (rv=2002) (sc=f769f6c0)

Dec 11 13:41:11 Rigel kernel: mptscsih: ioc0: attempting target reset! (sc=f769f6c0)

Dec 11 13:41:11 Rigel kernel: sd 1:0:1:0: [sdc] CDB:

Dec 11 13:41:11 Rigel kernel: cdb[0]=0x85: 85 06 20 00 00 00 00 00 00 00 00 00 00 40 e0 00

Dec 11 13:41:11 Rigel kernel: mptscsih: ioc0: target reset: FAILED (sc=f769f6c0)

Dec 11 13:41:11 Rigel kernel: mptscsih: ioc0: attempting host reset! (sc=f769f6c0)

Dec 11 13:41:11 Rigel kernel: mptbase: ioc0: Initiating recovery

Dec 11 13:42:04 Rigel kernel: mptscsih: ioc0: host reset: SUCCESS (sc=f769f6c0)

Dec 11 13:42:04 Rigel kernel: mptbase: ioc0: LogInfo(0x30030501): Originator={IOP}, Code={Invalid Page}, SubCode(0x0501) cb_idx mptbase_reply

Dec 11 13:42:04 Rigel kernel:  end_device-1:1: mptsas: ioc0: removing sata device: fw_channel 0, fw_id 4, phy 1,sas_addr 0x45763746cbb5af9c

Dec 11 13:42:04 Rigel kernel:  phy-1:1: mptsas: ioc0: delete phy 1, phy-obj (0xf771ec00)

Dec 11 13:42:04 Rigel kernel:  port-1:1: mptsas: ioc0: delete port 1, sas_addr (0x45763746cbb5af9c)

Dec 11 13:42:04 Rigel kernel: sd 1:0:1:0: [sdc] Synchronizing SCSI cache

Dec 11 13:42:14 Rigel emhttp: mdcmd: write: Invalid argument

Dec 11 13:42:14 Rigel kernel: sd 1:0:1:0: Device offlined - not ready after error recovery

Dec 11 13:42:14 Rigel kernel: md: disk1: ATA_OP e0 ioctl error: -22

Dec 11 13:42:14 Rigel kernel: sd 1:0:1:0: [sdc] 

Dec 11 13:42:14 Rigel kernel: Result: hostbyte=0x01 driverbyte=0x00

Dec 11 13:42:14 Rigel kernel: mptbase: ioc0: LogInfo(0x30030501): Originator={IOP}, Code={Invalid Page}, SubCode(0x0501) cb_idx mptbase_reply

okay - i have a drive that's PRETTY empty. how funny that for about a year i ran with it in place, not being used/assigned. about a week or two ago, i assigned it to the array. I can take the filesthat are on that, move it to another disk on the array, and do this. i'm safe to move within the array in this condition right?

You can move files around, but keep in mind, the longer you run with a degraded array, the better the odds of having another drive failure, at which point you will lose the contents of both failed drives.

 

You can NOT remove any of the current array drives, or all of the files on the failed drive will be forever lost. If you cannot source another drive, you could copy the files from the failed emulated drive onto the other array drives, then rebuild parity after setting a new config without the failed drive.

 

You are very close to losing the data on that drive, so be sure you understand exactly what you are doing before you do anything. If you do not have backups of everything on that drive, now is the time to make a copy elsewhere of the data that you want to keep, in case something else fails.

thanks again - the fear is starting to set in. I have 100% of the data on that drive in CrashPlan. but am hoping to avoid a 2TB restore :-)

 

so if i copy data from disk1 (failed) to disk6 (empty)... using the shrink procedure... i should be OK, right? thanks for the warning. i needed it. I almost pulled out disk6!

 

the drives that i have available ATM are all bigger than parity. and non a pre-cleared.

 

 

EDIT:

 

why does step 3 say: You should not encounter any duplicates any more.

 

 

 

so if i copy data from disk1 (failed) to disk6 (empty)... using the shrink procedure... i should be OK, right? thanks for the warning. i needed it. I almost pulled out disk6!

 

the drives that i have available ATM are all bigger than parity. and non a pre-cleared.

I'm not sure what you mean by "using the shrink procedure", but as soon as you have a copy of all the disk1 data elsewhere, you can set a new config and omit the drive that was in disk1 slot and generate new parity with the remaining drives.

tried to do the rename/move thing in MC ... moved from /mnt/disk1/Videos to /Videos_  then aborted that; and tried to move /Videos_ to /mnt/disk1/Videos ... stalled for a bit, but looks like it finished

 

i meant doing the array shrink process [http://lime-technology.com/wiki/index.php/Shrink_array] and kicking disk1 out of the array.

 

now moving data to disk6 ... going sorta slow, but from a dead drive? fast enough. going to let this go and step away for a minute.

 

Thanks for your help johnathanm - will report back in a bit

 

 

why does step 3 say: You should not encounter any duplicates any more.

Ahh, now I understand. When you copy files to an identical path on 2 different array disks, unraid can't resolve 2 identical file names in the user share, and it generates a duplicate file message in the system log. By renaming the root level folders, it's no longer an identical path, so no dupe messages. I wouldn't worry about it at this stage of the game. It's only relevant to user shares anyway, it has no bearing on disk shares.

why does step 3 say: You should not encounter any duplicates any more.

Ahh, now I understand. When you copy files to an identical path on 2 different array disks, unraid can't resolve 2 identical file names in the user share, and it generates a duplicate file message in the system log. By renaming the root level folders, it's no longer an identical path, so no dupe messages. I wouldn't worry about it at this stage of the game. It's only relevant to user shares anyway, it has no bearing on disk shares.

 

great ! i'm moving along. i can't believe that this data is coming from the emulated drive. I'm only sustaining 15MB/s on the move but who cares! i WISH i started mc with screen so that i dont' have to worry about a failed connection etc.

 

thanks - this will probably take a while. i had a FULL 2TB of data on that drive. so if my math works, im thinking 20hrs?

According to my math that's more like 37 hours. 

According to my math that's more like 37 hours. 

 

yup - but you know what? better than getting it out of crashplan at MAX of 12MB/s on my 101Mbit connection.

 

i still can't get over how well the emulated drive is performing.

Very often, drives that suddenly drop out may not be at fault at all.  Rather, there may have been an interface or controller issue.  Once you reboot, the drive should reappear, but UnRAID won't automatically add it back.  Since you've probably rebooted (you checked cables and power), try another SMART report from it.  Your drive and all its data may be completely fine.

Very often, drives that suddenly drop out may not be at fault at all.  Rather, there may have been an interface or controller issue.  Once you reboot, the drive should reappear, but UnRAID won't automatically add it back.  Since you've probably rebooted (you checked cables and power), try another SMART report from it.  Your drive and all its data may be completely fine.

 

Thanks Rob! didn't actually reboot, just checked to make sure cables and power are seated properly. When something fails, i'm VERY afraid to reboot. i know Unraid has my back covered (and now my faith shot WAYYYY up in the product). but my fear was more going wrong during the reboot.

 

as johnathanm mentioned unraid handle this very gracefully. so much so, that I couldn't even tell HOW i was seeing the folders etc until i saw his post. I knew about the emulated drive, but expected it to be dog slow, and be more like a data recovery type operation. i wouldn't be surprised if my family watched a movie while the array was in this faulted condition.

 

i'm GLAD i didn't fill my array to the gills, and had a completely empty, pre-cleared equivalent sized drive handy. I WILL forever now have a spare drive in the ready state for just this reason.

 

and if all this goes away after a reboot? well then i learned how nicely unraid handles this type of situation and now have practice for the inevitable failure in the future.

OK so the move appeared to be going VERY well.. .and i think there was only about 60GB left to move over... then i noticed THIS on the console (attached).

 

basically the file it's copying the source/target doesn't match... it looks like it's moving a .mkv file to a thumbnail?

 

I think the files got moved over. the free space on disk6 is 8GB, which is what disk1 was at before the fail.

 

edit: disk1 still shows 548GB in use... why would that be? i did the ren/move option in MC

I don't have an answer to your above question, but thought I would let you know how to get Putty to draw the MC screen correctly so it's not all garbled.

 

In the putty settings go to window -> translation, and change the character set to UTF-8.

thank you! i was wondering why it was doing that.

 

strange, it repeated that for a bunch of different source files... but should i worry about this? or is it just a display issue?

Very often, drives that suddenly drop out may not be at fault at all.  Rather, there may have been an interface or controller issue.  Once you reboot, the drive should reappear, but UnRAID won't automatically add it back.  Since you've probably rebooted (you checked cables and power), try another SMART report from it.  Your drive and all its data may be completely fine.

 

you were correct! i added in a new HBA and NIC - so I had to shutdown/reboot. drive is back.

 

now, do i trust it? should i run a pre-clear on it?

 

SMART report attached

smart_sdg.txt

Smart report looks fine. No reallocated sectors or pending reallocation sectors. I'd preclear it and see what happens.

Smart report looks fine. No reallocated sectors or pending reallocation sectors. I'd preclear it and see what happens.

 

thanks that's kinda what i gathered from seeing the wiki - appreciate the confirmation.

 

i'm running 3 rounds of pre-clear on it. i didn't get the latest version of the script, but should be fine, i think this one is from May.

now, do i trust it? should i run a pre-clear on it?

 

As he said, the SMART report shows that there is nothing wrong with the drive.  To be honest though, I cannot think of any reason why it would be useful to Preclear this perfectly good drive.  There is no evidence that the drive was at all at fault for its being dropped by your SAS controller.  Perhaps someone more knowledgeable can see a fault code related to the drive in your syslog fragment, but I can't.  It does show that Disk1 spun down without issue at Dec 11 08:24:33, but when it was directed to spin down a little later at Dec 11 13:40:36, the disk controller reports trouble roughly 30 seconds later.  It is able to stop the task, it is able to reset the host (the controller), but the attempt to reset the target (the drive) fails.  Since it is extremely unlikely that the drive would have received a normal disk I/O request just as it was directed to spin down (indicating no I/O at all recently), the only task it could have been working on was the spin down request.  So it appears that the drive must have completely lost contact a little before this, but there is not a single error or clue as to why.  And the SMART report shows that the drive saw nothing unusual to report either, it's clean.  Perhaps a cable fell off?  Don't know.  I don't understand why your pic shows 2 errors, but perhaps the second error occurs elsewhere in your syslog.

 

If you still want to Preclear it, there is no need to do it 3 times.  Multiple Preclears are designed to burn in a brand new drive, to try to get a marginal drive to fail early rather than when put into production.  The only other time you would want to Preclear it again would be to follow up on a problem found from a previous Preclear.

 

I would like to add that in my view the best practice after a problem like this would be to set the drive aside for awhile, because it is the original drive.  You may have copied all of its files elsewhere, but that location is still the copy not the original.  I would want to preserve the original drive untouched for awhile, until I was positive that all of the copy was intact and successful.  There is always the possibility of a bad copy somewhere, or a user error in the process, such as forgetting about or missing one little but important folder (that's something I might do, I'm so forgetful!).

now, do i trust it? should i run a pre-clear on it?

 

As he said, the SMART report shows that there is nothing wrong with the drive.  To be honest though, I cannot think of any reason why it would be useful to Preclear this perfectly good drive.  There is no evidence that the drive was at all at fault for its being dropped by your SAS controller.  Perhaps someone more knowledgeable can see a fault code related to the drive in your syslog fragment, but I can't.  It does show that Disk1 spun down without issue at Dec 11 08:24:33, but when it was directed to spin down a little later at Dec 11 13:40:36, the disk controller reports trouble roughly 30 seconds later.  It is able to stop the task, it is able to reset the host (the controller), but the attempt to reset the target (the drive) fails.  Since it is extremely unlikely that the drive would have received a normal disk I/O request just as it was directed to spin down (indicating no I/O at all recently), the only task it could have been working on was the spin down request.  So it appears that the drive must have completely lost contact a little before this, but there is not a single error or clue as to why.  And the SMART report shows that the drive saw nothing unusual to report either, it's clean.  Perhaps a cable fell off?  Don't know.  I don't understand why your pic shows 2 errors, but perhaps the second error occurs elsewhere in your syslog.

 

If you still want to Preclear it, there is no need to do it 3 times.  Multiple Preclears are designed to burn in a brand new drive, to try to get a marginal drive to fail early rather than when put into production.  The only other time you would want to Preclear it again would be to follow up on a problem found from a previous Preclear.

 

I would like to add that in my view the best practice after a problem like this would be to set the drive aside for awhile, because it is the original drive.  You may have copied all of its files elsewhere, but that location is still the copy not the original.  I would want to preserve the original drive untouched for awhile, until I was positive that all of the copy was intact and successful.  There is always the possibility of a bad copy somewhere, or a user error in the process, such as forgetting about or missing one little but important folder (that's something I might do, I'm so forgetful!).

 

thank you for the detailed reply. it probably was a bad idea to fire off 3 full cycles of preclear. i guess after reading that one line in the wiki about it NEVER being a fluke, i mentally sealed the drive's fate as faulty. though your earlier reply should have changed that!

 

okay - can i kill the pre-clear after the first cycle?

 

and you are right about setting it aside until you can confirm the copy was good. I think I got a very solid confirmation... Crashplan recognized that drive 1 is missing; then when i added drive6 to the backup routine, it did all the differencing/synchronizing, etc that it does - and nothing really uploaded.

 

i can't say it enough though - Unraid, handled this with awesome svelte. it helps that the data here wasn't critical, can be replaced (with considerable time/effort though). but i got a VERY good trial run here. I was always tempted to pull a cable, or a drive out of it's cage to see what happens, but never had the guts to do it. now i know exactly what happens and how to deal with it. hopefully next time, it will be this smooth.

 

 

 

 

and you are right about setting it aside until you can confirm the copy was good. I think I got a very solid confirmation... Crashplan recognized that drive 1 is missing; then when i added drive6 to the backup routine, it did all the differencing/synchronizing, etc that it does - and nothing really uploaded.

That's great!

 

after reading that one line in the wiki about it NEVER being a fluke

Just curious, where in the wiki was that?

 

okay - can i kill the pre-clear after the first cycle?

Without looking, I'm pretty sure that Preclear can be canceled, but you have to be very careful where you cancel it.  The Pre-read and Post-read phases should be safe to cancel as they should only be read-only testing.  But cancel it anywhere else and the drive will probably be unusable until re-prepped.  One blow from the zeroing phase and the drive is unusable.  I believe Preclear has the options to re-prep it quickly though, but I'm rusty on how.

That's great!

hopefully it's accurate in it's assessment.

 

Just curious, where in the wiki was that?

 

this line right here... link below. there was a nother one, that unraid doesn't mark a red lightly. sooo given the warnings, i was prepping for a warranty return (this dive has only been in service 6 months or so) so i figured sill within the bathtub curve of failure rates.

 

"Okay, the cable was loose, or I think the failure was a fluke ( note: it is NEVER a fluke )-"

 

http://lime-technology.com/wiki/index.php?title=Troubleshooting#Re-enable_the_drive

 

Without looking, I'm pretty sure that Preclear can be canceled, but you have to be very careful where you cancel it.  The Pre-read and Post-read phases should be safe to cancel as they should only be read-only testing.  But cancel it anywhere else and the drive will probably be unusable until re-prepped.  One blow from the zeroing phase and the drive is unusable.  I believe Preclear has the options to re-prep it quickly though, but I'm rusty on how.

 

will look into it before stopping.

the drive is fine - it *was* a fluke. by the time i got back on the console, it had already started the 3rd cycle - so i just let it ride.

 

Passed that well. re-added to the array, ran a parity check... everything is all well. marking this as solved. Thanks everyone for the tips.