This morning during the monthly parity check - I had a drive redball on me. It wasn't a drive I was expecting to see fail - it's a WD 5Tb Red that's only about 18-24 months old...SMART report indicates a pass - so I'm just wondering what my best step would be - ideally I can rebuild the data and continue on with the drive - but It would be good to get some knowledgeable people's opinions...

 

Diagnostic file is attached.

 

Happy New Year to all!

tower-diagnostics-20180101-0936.zip

Check your cabling (SATA and Power) and the SMART values for CRC-Errors

Edited January 1, 20188 yr by Zonediver

Thanks for the answer. The server has been running for some time without issues and I haven't been in it to knock any cables. Unless the cables have spontaneously failed, they should be fine...will look at CRC errors.

Like Zonediver mentioned the disk shows an extremely high number of CRC errors, most likely a very bad SATA cable:

 

199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       31896

You also have several ATA errors on your SSD, check/replace that cable also.

Thanks both for the answers - still makes no sense that the cable is suddenly in bad shape - that's one of the things that confuses me. I see what you both mean about the high CRC count. I have a couple of spare sff8087 to SATA cables so will see if I can swap these out then run a new check.

A badly seated cable can start to misbehave on it's own because the tiny vibrations from fans and disks can work dust in under the connectors. There is also the possibility that a cable is touching a sharp metal surface.

49 minutes ago, pwm said:

A badly seated cable can start to misbehave on it's own because the tiny vibrations from fans and disks can work dust in under the connectors. There is also the possibility that a cable is touching a sharp metal surface.

That would make sense...I am replacing the cable anyway as I have a brand new spare - Will have to make sure they latch properly...

So I replaced the cable and did a rebuild onto the drive - think I must have unseated a different cable as I now have a different disk in an error state and disabled with hundreds of thousands of read errors - that was working perfectly well before I went into the case...I need a bigger case - squeezing my 17 hard drives into a Fractal Design Define XL is a bit too tight for comfort!

Or use Drive Cages, so you don't have to open up the case to change out drives.  You also should be aware of this issue since you have some WD drives.

 

          https://support.wdc.com/knowledgebase/answer.aspx?ID=10477

Thanks for the info Frank - always good to know. I have drive cages where I can, but most of the drives are internal - I am hoping to save and get wife approval for a Norco 24-bay case or similar with all cages...

 

Thinking about this - if I had a mass of new read errors on a different drive (No5) on my drive rebuild (No11) what does that mean for the drive rebuild data? Parity was valid before I rebuilt the data

16 minutes ago, aspdend said:

if I had a mass of new read errors on a different drive (No5) on my drive rebuild (No11) what does that mean for the drive rebuild data?

It means the rebuilt disk will have corrupt data

Thats what I am afraid of. So should I power down, reseat the cables and do another rebuild onto disk11 - assuming this time that the connections are fine and that disk 5 won't throw up more read errors?

Yes

4 minutes ago, aspdend said:

Thats what I am afraid of. So should I power down, reseat the cables and do another rebuild onto disk11 - assuming this time that the connections are fine and that disk 5 won't throw up more read errors?

Yes.

10 minutes ago, pwm said:

Yes

Yes.

Thanks = thats what I thought

OK - so with a reboot - my disk 5 is ow coming up as unmountable - not sure the best way to go forward with this. Disk 11 was the original issue with what looks like a cable issue. replacing that seems to have solved that issue, but I must have knocked another cable as disk 5 threw up ridiculous numbers of read errors during the disk 11 rebuild. So disk 11 has rebuilt with corrupt data and needs to be rebuilt, but disk 5 is also now needing a rebuild? or some other treatment...Any gurus got any ideas?

If the disk is unmountable on need to run xfs_repair:

 

https://lime-technology.com/wiki/Check_Disk_Filesystems#Drives_formatted_with_XFS

Ok, so do an xfs repair on disk 5 then hopefully should be able to rebuild the data on disk 11...let me see what I can do

right - I tried the xfs_repair and no go - got this message

 

ERROR: The filesystem has valuable metadata changes in a log which needs to be replayed.  Mount the filesystem to replay the log, and unmount it before re-running xfs_repair.  If you are unable to mount the filesystem, then use the -L option to destroy the log and attempt a repair.

 

Tried to mount the drive - but nothing happening at all so eventually went with the -L option and this seems to have done the trick - will check in Lost+Found later as something has been put in there I see from the logs. Sp now my server is back up and running, however, disk 5 is not online - this is emulated. How do I get 5 back online so I can rebuild disk 11? 

I thought disk5 was unmountable but online, unRAID won't disable 2 disks with single parity, or was it disable after disk11 finish rebuilding? In doubt and if you havn't rebooted yet post your diagnostics.

 

1 hour ago, johnnie.black said:

I thought disk5 was unmountable but online, unRAID won't disable 2 disks with single parity, or was it disable after disk11 finish rebuilding? In doubt and if you havn't rebooted yet post your diagnostics.

Disk 5 was disabled after disk 11 finished the rebuild. I have rebooted after the rebuild - here are the latest diagnostics if they are any good...

tower-diagnostics-20180103-0034.zip

Looks like disk5 problems aren't limited to a cable issue:
 

  5 Reallocated_Sector_Ct   0x0033   099   099   036    Pre-fail  Always       -       1152
183 Runtime_Bad_Block       0x0032   099   099   000    Old_age   Always       -       1
187 Reported_Uncorrect      0x0032   097   097   000    Old_age   Always       -       3
197 Current_Pending_Sector  0x0012   100   100   000    Old_age   Always       -       8
198 Offline_Uncorrectable   0x0010   100   100   000    Old_age   Offline      -       8

Your options are limited, I would probably replace disk5, then try to copy everything possible from the old disk5, you«ll still be left with corruption on disk11, but not much else you can do, IMO dual parity is recommended for an array of that size.

55 minutes ago, johnnie.black said:

Looks like disk5 problems aren't limited to a cable issue:
 

  5 Reallocated_Sector_Ct   0x0033   099   099   036    Pre-fail  Always       -       1152
183 Runtime_Bad_Block       0x0032   099   099   000    Old_age   Always       -       1
187 Reported_Uncorrect      0x0032   097   097   000    Old_age   Always       -       3
197 Current_Pending_Sector  0x0012   100   100   000    Old_age   Always       -       8
198 Offline_Uncorrectable   0x0010   100   100   000    Old_age   Offline      -       8

Your options are limited, I would probably replace disk5, then try to copy everything possible from the old disk5, you«ll still be left with corruption on disk11, but not much else you can do, IMO dual parity is recommended for an array of that size.

Johnnie - thanks as usual for your help and knowledge - the sad thing is that I have a new 8Tb drive ready and waiting to swap out for one of my old 2Tb WD Red's and disk 5 was one of the candidates - I was going to do that after the parity check....I will be going for the dual parity - I need to sort out a new case suitable for up to 24 drives - but have a plan for that now that the wife has agreed to...So should I just pull disk 5 and replace with the new drive so that it rebuilds onto the new drive and then see what I can pull off of the old disk 5 and copy anything I can across to the new disk 5? Then see what data I can determine is right with Disk 11? It should be mainly just media either from TV or disks so easily replaceable...

53 minutes ago, aspdend said:

So should I just pull disk 5 and replace with the new drive so that it rebuilds onto the new drive and then see what I can pull off of the old disk 5 and copy anything I can across to the new disk 5?

Yes, this is what I'd do, rebuild to a new disk, there will be corrupt files due to previous errors, then copy everything you can from the old disk and overwrite rebuilt files, every file you can copy should be considered OK, thus leaving you with hopefully just a few corrupt/in doubt files.

 

56 minutes ago, aspdend said:

Then see what data I can determine is right with Disk 11?

Yes, this would be easy with checksums or btrfs, without it you'll need to test each file.

Just now, johnnie.black said:

Yes, this is what I'd do, rebuild to a new disk, there will be corrupt files due to previous errors, then copy everything you can from the old disk and overwrite rebuilt files, every file you can copy should be considered OK, thus leaving you with hopefully just a few corrupt/in doubt files.

 

Yes, this would be easy with checksums or btrfs, without it you'll need to test each file.

OK - that sounds like a plan, will try this later when I get home. How would one use the checksums to determine file corruption?

They'd need to be created before the problem, btrfs creates them automatically on write, with xfs you'd need an external utility to create them manually, by using the Dynamix file integrity plugin or corz for example.