Backstory leading to issue:

After a flawless unraid run for years with no disk issues I decided to do some server upgrades and man did the dominos start to fall. I lost 2 hard drives one rebuild after another, luckily the first one happened as I was adding in a second parity drive so it was just emulated and it finished rebuilding so I had two parity drives from that point on. All these failures I think is leading to my issue, some files have even gone missing but not worried on recovering them as they are easily replaceable. 2 other hard drives were starting to show errors during rebuilds so I replaced them as well. One had over 32,000 pending sector errors during a rebuild but unraid never kicked it out of the array, think the issue started right after this happened / I replaced that drive. Which was disk 9 in the array. 

 

Issue at hand:

(Currently running version 6.12.3) After that last drive failure (I think) all the shares vanished off of the shares tab and are no longer accessible on the network. I can see the data on the disks and all the dockers seem to have no issue accessing the data and are functioning properly. 

 

Things I tried: 

1. Reboot - No change

2. Upgrade OS from 6.12.1 to 6.12.3 - No Change

3. Removed some old incompatible apps - No Change

4. Upgraded all dockers and applications - No Change 

 

The server is a Supermicro 846 chassis with all drives connected via a backplane so I don't think cable / power issues are to blame here. Plan is to get rid of these old smaller drives and migrate everything over to the 14tb drives as I add in more. Rebuilds were slowing down to 500k at times and only averaged around 45mb/s so there is a definite slowdown on one or more of the drives.

 

Let me know if you need more info as I'm sure I forgot something important, 

 

Thank You!

tower-diagnostics-20230723-0914.zip

Your syslog is flooded with:

Quote

XFS (md6p1): Metadata corruption detected at xfs_buf_ioend+0xac/0x384 [xfs], xfs_inode block 0x67e8100 xfs_inode_buf_verify
XFS (md6p1): Unmount and run xfs_repair
XFS (md6p1): First 128 bytes of corrupted metadata buffer:

 

So disk 6 is having an issue and needs xfs_repair run. 

 

When disk 6 is fixed the array can start and you will see the dockers. 

Ahh corruption, that does make sense as I did have more reboots in the past couple weeks than i have in the past two years. Sometimes the array wouldnt shut down properly either so that's probably where the corruption came from. 

 

Started the array in Maintenance Mode and ran the xfs repair, started her back up and it worked! Wouldn't have imagined the file system would have corrupted, surprized the drive was still mounting bad as it was.

 

Thank you for the help, no way I would have figured that out on my own