[6.11.5] Disk 9 disabled after 1175 errors

[mrbens]

Need some assistance please.

 

Disk 9 has been disabled after 1175 read and write errors.

 

SMART not looking good.

 

 

Attached diagnostics.

 

Does it look like it'll need binning, or is it worth replacing the SATA cable with a new one and trying a preclear to re-add it?

 

Thanks.

 

 

 

tower-diagnostics-20230104-1035.zip

[JorgeB]

Disk appears to be failing, you can run an extended SMART test to confirm, if it fails replace it.

[itimpi]

Do you have notifications set to let you know about errors so you can take action before errors become unrecoverable?   With that many reallocated sectors I would have expected you to have been getting notifications for some time.

[mrbens]

Thanks JorgeB. I've got an extended test running now.

 

itimpi just realised I had notices and warnings disabled and just had notifications for OS updates and alerts. I've enabled them now and worryingly it looks like disk 6 is also having issues with reallocated sectors.

 

 

 

I'm currently running rsync -avPX /mnt/disk9/ /mnt/disk1/ to do New Config to remove disk 9 as I've got space on disk 1 for now, but worried about doing that in case disk 6 fails and would leave me without parity to rebuild it.

 

Disk 10 now too:

 

 

 

Currently up to 592 reallocated sectors 30 minutes later.

 

No new errors on the Main page.

 

Attached new diagnostics.

 

What would you recommend please?

 

rsync speed has also dropped to under 3MB/s while running a SMART extended test of disk 9 and short test of disk 6.

 

 5,150,177,280 100%   25.63MB/s    0:03:11 (xfr#782, ir-chk=1052/2045)

  8,468,856,832  46%    2.60MB/s    1:01:37

 

It was running about 25MB/s earlier, and I think over 50MB/s previously when copying a different disk to convert to xfs a few days ago when no disks were disabled. I can understand why it's slower now with a disabled disk, but should the SMART checks be slowing it down so much from 25MB/s to 3MB/s? Short SMART seems to be stuck on 90%. Holding off running a test on disk 10 while the server is busy.

 

Once I've finished converting all my disks to xfs I'm adding a second parity as I've had issues with a few disks recently.

tower-diagnostics-20230105-0529.zip

Edited January 5, 2023 by mrbens

[mrbens]

Stopped the short SMART test of disk 6 and the rysnc speed improved again.

 1,211,924,480  11%   23.21MB/s    0:06:17

 

After disk 9 has finished copying to disk 1, I could copy disk 10 and then disk 6 to disk 1 too. Disks 6, 9 and 10 are only small old 3TB and 2TB luckily, so don't mind removing them from the array in case they die soon.

 

Disk 10 now on 760 reallocated sectors. Disk 6 hasn't increased from 6304.

 

Stopped the extended test too and rysnc speed briefly went as high as 49MB/s but then stayed around or under 18MB/s on big files since then.

 

For peace of mind I'd rather copy off all the data quicker and can test them later when they are not part of the array. Had 2 disks fail at the same time before and lost one of them completely, the other I could save a lot with ddrescue but it took months and had a lot of corrupt files.

Edited January 5, 2023 by mrbens

[JorgeB]

With multiple disks possibly failing I would start by backing up the most important data somewhere outside the array.

[mrbens]

Managed to move most data off the 3 disks. Luckily didn't lose anything important. It was getting a bit scary with all the errors on disk 6 which made it slow to copy the data off disks 6 and 9! Probably got some corrupt files, but luckily the disks are in a share of unimportant files.

 

Currently trying to erase and clear them to dispose of, but 2 of them are taking a very long time due to the errors and keep pausing temporarily. Top one says 35MB/s but only does that for a few seconds before pausing for a while.

 

Is it worth skipping the pre and post read on them, or is it recommended to leave them all to finish normally?

 

The bottom disk is erasing at a really slow speed. Guess there's nothing I can do about that.

 

 

 

 

 

Edited January 16, 2023 by mrbens