5million+ read errors + I/O Errors after Rebuild

[silasfelinus]

I've replaced two hard drives in approximately two weeks, and yesterday a third disk died.  I started a data rebuild, but had 5mil+ read errors on a different Disk 12 and a flood of I/O errors "xfs_repair: read failed: Input/output error can't read data block 0 for directory inode 3180500740 error 5" on the Disk 9 that is supposed to hold the emulated data.

 

I stopped my data-rebuild (to Disk 9) at 45% this morning after the errors on Disk 12 appeared. I ran short smart tests on everything, which found the I/O errors on 9 and would not let me complete the xfs-repair.  Possibly regretably: there was a jammed log on 12, and I saw the warning that I could lose "valuable metadata" but I deleted the metadata and had it complete xfs-repair.

I'm just ran the disks in maintenance mode with a disk check, but it said it would take 3 days to complete, and those error messages on 12 re-appeared. I stopped the test, and am now running an extended self-test on Disk 12.

Any advise is appreciated.

alexandria-diagnostics-20230804-0809.zip

[JorgeB]

It's not logged as a disk problem, looks more like a power/connection issue, if it's been happening to multiple disks see if they have something in common, could also be the PSU.

[silasfelinus]

12 minutes ago, JorgeB said:

It's not logged as a disk problem, looks more like a power/connection issue, if it's been happening to multiple disks see if they have something in common, could also be the PSU.

That tracks! I just checked cabling. Disk 9 and Disk 12 were connected on the same strand next to each other. I've got my NAS maxed with 16 drives in a 15 drive case (plus 4 SSDs), and cabling is a bit of a challenge. I even disconnected the cabling around 12 when I put in the last drive, and I'm  fairly certain I didn't put that section back in the exact same cable layout. I'm waiting on the reboot and a fresh report....

 

Thank you for the response, it's renewed a smidgen of hope.

[silasfelinus]

I think we're good! 

 

I was worried when I switched cabling and the new drive was suddenly "unmountable". Instead, I started in maintenance and ran short smart tests on all the drives, Disk 9 took a while but finished (for the first time successfully), and now the drive is being rebuilt without errors (so far), and I'm even seeing data that went missing last night and I'd already written off.

 

Thank you for the help! You saved my day!