UNRAID disabled "good disks" | Failed rebuild | Corrupted Disk | Problems keep escalating

[Husker_N7242C]

Hi guys, I've had a really unusual set of circumstances result in what appears like multiple drive failures (but isn't). I've probably lost 2TB of data, I'm at risk of loosing 8TB more and have no partiy now. Please help if you can (diagnostics attached)

Order of events:

1. disk2 was disabled by UNRAID for smart errors (7 year old 2TB)

2. Replaced disk2 with new Ironwolf 8TB

3. rebuild failed for some reason and the parity disk was disabled by UNRAID. The parity is a quite new 8TB ironwolf also with no SMART errors to date.
4. I tried to re-seat the cables and reboot just incase something came loose

5. After reboot disk2 appears to be rebuilt which is impossible in the short time it was rebuilding

6. Parity drive is still disabled and stopping the array and trying to remove and re-add it doesn't work, UNRAID doesn't want it.

7. I tried removing disk2 in the GUI because it is obviously corrupt. UNRAID tries putting data on it but the data just vanishes into oblivion.

8. Ran a read check on all drives hoping that UNRAID would see that disk2 is corrupt and let me do something with it but it got 0 errors from disk2

9. disk4 got 120,000 read errors from the test.... another near new ironwolf 8TB with no smart errors which was fine until now.

 

I don't even know where to start with this. I realise that 2TB from disk2 is probably gone. I can live with that I guess. I really don't want to loose another 8TB from disk4.

Any help would be received gratefully.

nas-diagnostics-20200812-1631.zip

[JorgeB]

Unfortunately only thing visible in the syslog is filesystem corruption on disk2, that caused the syslog to be spammed and there are many missing hours, and no disk errors visible.

 

Multiple disk errors in apparently healthy disks suggest a controller/power/connection problem, but difficult to guess without the syslog showing the issues.

 

I would reboot to clear the log and re-sync parity, if more error post new diags.

[Husker_N7242C]

Thanks tee-tee. I really appreciate the reply.  I've rebooted and this time it has let me put the parity drive back and is trying a sync.
I don't think that disk2 can possibly have rebuilt correctly though. The rebuild failed after like 2 hours.
If this parity sync finished I'm not sure I can trust that I don't have a corrupt allocation table or something on disk2.

I've attached a fresh diagnostics in the hopes that it will now contain something useful?

nas-diagnostics-20200812-1906.zip

[JorgeB]

1 hour ago, Husker_N7242C said:

I don't think that disk2 can possibly have rebuilt correctly though. The rebuild failed after like 2 hours.

Most likely, unfortunately nothing of the rebuild is on the diags, you need to check the data, that's why it's good to have checksums of all files.

 

1 hour ago, Husker_N7242C said:

I've attached a fresh diagnostics in the hopes that it will now contain something useful?

Just the everything looks normal so far.

[Husker_N7242C]

Thanks again Johnnie (sorry I wrote tee-tee earlier, I was on my mobile and mis-read).

I attempted another parity sync but it failed and disabled the parity drive again (new diag attached).

 

Re: checksums, I do have Dynamix File Integrity Plugin installed. It runs monthly with SHA2. I have no clue how to use it to help my situation? I've attached a screenshot as it shows that disk2 "build" and "export" are not up to date

nas-diagnostics-20200813-1709.zip

[JorgeB]

Doesn't look like a disk issue, maybe power/cables, if you have spare cables and/or PSU try that.

 

27 minutes ago, Husker_N7242C said:

I do have Dynamix File Integrity Plugin installed.

You can run a check on disk2, but firt you need to run xfs_repair on it.

[Husker_N7242C]

Thanks. I know the parity is on a PCIe HBA card in IT mode (2x SAS to 8x SATA). I did swap the SATA end from the SSD that is unassigned to the Parity which made no difference. I'll have a look and see which HDDs are on the HBA and see if there is a pattern.

[Husker_N7242C]

I've ended up running the pre-clear script on both drives with pre and post read cycles. Both passed no errors. I've added disk2 back to the array and copied 2TB of data back to it without an error. I'll add the parity back tomorrow and rebuild.

I saw a post in the Facebook group of (what looks like) the same thing happening to others. Parity gets disabled, nobody can work out why, they check the disk and put it back and all is good. Maybe it is a bug or UNRAID is disabling the drive too ruthlessly? 

[JorgeB]

3 hours ago, Husker_N7242C said:

Maybe it is a bug or UNRAID is disabling the drive too ruthlessly? 

Unraid doesn't disable a disk without a write error, write errors can be caused by many things, post new dags if it happens again, hopefully it will catch the issue next time.