Shares missing after failed HDD's

[driftking]

Hello, I have been a long time unraid user with no major issues until now. I am having a bit of a head scratcher here. On Feb 1st at the end of the monthly parity check, 2 HDD's were disabled and showed read errors. I took a screenshot and immediately did a safe shutdown. I received replacement hard drives the next day.

 

While the first hard drive replacement was rebuilding, another hard drive failed during the rebuilding process. This hard drive ended up with a whole bunch of read errors. 

 

I do have dual parity drives but my concern is that since another hard drive failed during the rebuild with already two drives failed, wouldn't that mean the array was unprotected? But, why are they shares not showing up any more?

 

I am now rebuilding the last failed hard drive, if you view the folder contents through the GUI, it looks like everything is still there, or is that not true and the folder contents is just a cached drive directory being shown?

 

Quick rundown of the sequence of failure and rebuild

1. Feb 1st monthly parity check (automatic). Received an email stating parity check failed with two drives showing read errors.
2. HDD-10 and HDD-14 failed with read errors.
3. safe shutdown of server to prevent any further loss.
4. Replaced HDD-10 and started rebuild. (Shares were showing up as normal, nothing alarming yet)
5. During the rebuild HDD-9 displayed errors on the main GUI screen, the errors kept counting up but the HDD-10 rebuild did complete.
6. Replaced HDD-9 and rebuild (Shares no longer show up......)
7. Replaced HDD-14 and rebuilding <----- in progress currently. (Shares still not showing up.....)

 

The System Log is filled with this error, but I don't know what to do or what it means. Is this a result of a 3rd drive failing during the 1st rebuild?

Quote

Feb 6 20:35:08 Tower kernel: XFS (md9): metadata I/O error in "xfs_imap_to_bp+0x5c/0xa2 [xfs]" at daddr 0x117eaf4e0 len 32 error 117 Feb 6 20:35:08 Tower emhttpd: error: get_filesystem_status, 6102: Structure needs cleaning (117): scandir Structure needs cleaning

 

 

tower-diagnostics-20220206-2034.zip

[JorgeB]

8 hours ago, mafiakid said:

XFS (md9): metadata I/O error in "xfs_imap_to_bp+0x5c/0xa2

Check filesystem on disk 9.

[driftking]

I definitely have data loss on the system due to the 3rd drive failing during the parity rebuild. Disk 10 finished rebuilding this morning but it didn't switch over like normal, it completed the rebuild without errors but stayed as un-mountable. 

 

 

I am currently trying to rebuild disk 8 and 10 again, i'm not sure what else to do. Even with the 2 parity drives, i still ended up in the worse case scenario.

[JorgeB]

11 hours ago, mafiakid said:

I definitely have data loss on the system due to the 3rd drive failing during the parity rebuild.

That's expected, do you still have the old disks? Might still be able to salvage some data that way.

[JorgeB]

2 hours ago, JorgeB said:

That's expected

To expand a little more on this, diags you posted don't show the rebuilds, but:

 

On 2/7/2022 at 4:36 AM, mafiakid said:

During the rebuild HDD-9 displayed errors on the main GUI screen, the errors kept counting up but the HDD-10 rebuild did complete.

This means that the rebuilt disk10 will be corrupt, since when there were read errors on disk9 writes to disk10 would be skipped, so disk will be empty in those sectors.

 

On 2/7/2022 at 4:36 AM, mafiakid said:

Replaced HDD-9 and rebuild

This would also result on a corrupt rebuild since disk10 was already corrupt.

 

On 2/7/2022 at 4:36 AM, mafiakid said:

Replaced HDD-14 and rebuilding

Same.

[driftking]

Thanks for the info. I still have two of the old disks, but unfortunately due to the 3rd failed disk during the rebuild, I lost a whole disk. 

 

I ended up removing all the drives from the server and bought new 8TB hard drives and now moving the data back to the server.