6.3.5: Disk Died; Replaced; ParityChk; 166k Errors on Other Disk

[wheel]

The sync (vs disk) correcting parity check was a total brain fart on my end, and I'm hoping it turned out okay (no error messages but I'll go back to check the underlying data as soon as I can).

 

I was just writing to disk 12 and the GUI threw up a read error, so I immediately pulled diagnostics to send here.

 

I have a precleared 8tb spare ready to replace Disk 12's 6tb, and I'm leaning towards just shutting down and throwing that thing in there to start a Disk 12 rebuild/upgrade now - any reasons I shouldn't do that in terms of better-safe-than-sorry?

 

Thanks for all the guidance!

tower-diagnostics-20200604-1054.zip

[JorgeB]

I would firs run an extended test on disk12, if it fails replace now, if it passes give a second change, there are also some risks doing a rebuild, e.g. if other disk fails.

[wheel]

OK, running extended test now - hate that it's consistently throwing up errors and need to replace a 6tb soon anyway, but definitely don't want to throw out disks unnecessarily during what could be a weird economic time for getting new disks.

 

Thanks for the quick response!

[wheel]

Soooooo something may be up with the Disk 12 slot.

 

That 6tb couldn't finish an extended smart test, so I dropped what I was pretty sure was a fine 8TB (precleared and SMART ok after being used in another box for a couple of years) into the slot for the rebuilt.  Had a choice between using an SMR Seagate and CMR WD and used the WD. Rebuild was interestingly exactly 75% complete (right at the 6tb mark) and the new 8tb in the Disk 12 slot started throwing up 1024 read errors and got disabled.

 

My instinct's to throw another 8TB spare in the slot and try it again, but something feels weird, so here's the diagnostics.  Am I reaching a point where something's likely wrong with the hotswap cage and I'm going to need to buy / replace that whole thing again?

tower-diagnostics-20200605-0534.zip

[JorgeB]

Looks like a connection/power issue, do you have another slot you could try?

[wheel]

Unfortunately not - it's an old box (first built in 2011, I want to say?), four 5-slot Norco SS-500 hotswap cages stacked on each other in the front.  Nothing ever really moves around behind the cages, and the only cable movement that I can recall since I first built it was when unplugging/replugging the cage's breakout cables when replacing the Marvell cards with LSIs back in December (and these issues with disk 12 started occurring maybe a quarter of a year later).

 

The hotswap cage containing Disk 12's slot is the second up from the bottom, and could be a massive pain to replace (presuming I can find a replacement of such an old model, or one that doesn't mess up the physical spacing of the other 3 hotswap cages).

 

Edit 2: any chance the rebuild stopped at *exactly* 6tb could be significant? Feels like a bizarre coincidence.

Edited June 5, 2020 by wheel

[JorgeB]

29 minutes ago, wheel said:

Edit 2: any chance the rebuild stopped at *exactly* 6tb could be significant?

Don't see how.

[wheel]

Yeah, I'm just reading tea leaves at this point and hoping there's something obvious I'm missing.

 

I have at least two (could be three in a couple of days) theoretically fine 8TBs ready to roll, and the original 6tb that was throwing up errors (which may have nothing to do with the disk, now) before the rebuild.

 

GUI shows the rebuild ("Read-Check" listed) as paused.  I'm guessing my next steps without a free slot to try are going to be:

 

Cancel rebuild ("Read Check").

Stop array, power down.

Place (old 6tb? another different 8tb?) into Disk 12 slot.

Try a rebuild again today (since I'm guessing unraid trying to turn the old 6tb into an 8tb but failing mid-rebuild means I can't simply re-insert the old 6tb and have unraid automatically go back to the old configuration?)

 

Any reasons why I shouldn't other than the fact that I'm playing with fire again with another disk potentially dying while I'm doing all these rebuilds?  I'm starting to think my only options are firedancing or waiting who knows how long for an appropriate hotswap cage replacement and crossing my fingers that I'll physically rebuild everything fine (and I'm almost more willing to lose a data disk's data than risk messing up my entire operation).

Edited June 5, 2020 by wheel