Ramshackleton

Update #2: All affected drives now use Molex -> SATA adapters. All drives recognized, no clicking detected. Parity rebuild underway, looking MUCH better in terms of ETA, closer to the day-ish timeframe I was expecting. Hopefully this is the end of the saga. Thanks again for all the help!

Update: Tonight, after fiddling around a bit, I decided to rip out my new power supply and replace it with the old one. After all, it's power-related, right? Long story short: drives started not showing up upon boot, some of the same ones between power supply changes. Then I realized what the problem was: these were shucked drives that previously were probably connected via a Molex -> SATA converter, but now were straight SATA from the PSU. But they were never fixed to deal with the 3.3 V pin issue. This probably wasn't my original problem, as I'd think you'd either have power or not, so it wouldn't be the problem causing the clicking/power drops. Just goes to show what changing too many things at once can do to really complicate matters. I'm too tired to address it tonight, but I'll work on it in the morning/afternoon tomorrow to see if I can get it fixed. @trurl thanks again for all your help thus far.

Thank you! I'm going to re-assess and revert back. Thank you so much for your help so far! Can't tell you how much I appreciate it.

Corsair RM750x. I had another Corsair model before, also 750W, it was just older and didn't have modular plugs. The problem here is that I have an untested and poorly-thought-out power scheme and doesn't ship with nearly enough connectors for this many drives. In any case I will clean this up ASAP.

You know, I think you're onto something - all these problems started after I moved the guts of this server to a new case/power supply. I thought this would be better than my old one, which was full of molex -> SATA power adapters. This one though does have a fair # of splitters - the drive in question is running on the same line as at least 4 other drives and several fans, might be even more drives, it's a bit of a mess. I'm going to try to clean this up and will report back. But in the meantime, I assume I'm safe to abort the parity build so I can shut down?

Ok, so LGTF is the one I said was clicking (and I can hear it doing it now as well). Note this was NOT the original parity drive. This was a data drive. It is mounted but apparently unhealthy. Is he the one slowing things down? Should I try to salvage what's on it and replace?

That is pretty much what I see, but the ETA on the parity rebuild is still really long, currently hovering around 100 days. Diagnostics attached. tower-diagnostics-20220722-1427.zip

Ok, I thought it was started when I took those diagnostics. It's been running all night. Just took the attached ones, thanks! tower-diagnostics-20220722-1129.zip

Ok, I removed the LWTC drive, the old parity disk, diagnostics attached. tower-diagnostics-20220721-2236.zip

I don't have a lost+found share, no. Data looks ok, but hard to know without a deep dive. I guess the question now is, am I safe to assign my new drive as parity, or will it try to take 255 days again?

It mounted this time! Diagnostics attached. tower-diagnostics-20220721-1508.zip

Sorry for being so thick guys, thanks for all the help. Here's the output of the -L, and new logs. Phase 1 - find and verify superblock... Phase 2 - using internal log - zero log... ALERT: The filesystem has valuable metadata changes in a log which is being destroyed because the -L option was used. - scan filesystem freespace and inode maps... sb_fdblocks 121671163, counted 123624614 - found root inode chunk Phase 3 - for each AG... - scan and clear agi unlinked lists... - process known inodes and perform inode discovery... - agno = 0 - agno = 1 - agno = 2 - agno = 3 - agno = 4 - agno = 5 - agno = 6 - agno = 7 - process newly discovered inodes... Phase 4 - check for duplicate blocks... - setting up duplicate extent list... - check for inodes claiming duplicate blocks... - agno = 0 - agno = 2 - agno = 5 - agno = 4 - agno = 6 - agno = 7 - agno = 1 - agno = 3 Phase 5 - rebuild AG headers and trees... - reset superblock... Phase 6 - check inode connectivity... - resetting contents of realtime bitmap and summary inodes - traversing filesystem ... - traversal finished ... - moving disconnected inodes to lost+found ... Phase 7 - verify and correct link counts... Maximum metadata LSN (1:1879329) is ahead of log (1:2). Format log to cycle 4. done tower-diagnostics-20220721-1350.zip

So just with -v? I did that and got this: Phase 1 - find and verify superblock... - block cache size set to 1473832 entries Phase 2 - using internal log - zero log... zero_log: head block 1879314 tail block 1879310 ERROR: The filesystem has valuable metadata changes in a log which needs to be replayed. Mount the filesystem to replay the log, and unmount it before re-running xfs_repair. If you are unable to mount the filesystem, then use the -L option to destroy the log and attempt a repair. Note that destroying the log may cause corruption -- please attempt a mount of the filesystem before doing this.

Phase 1 - find and verify superblock... - block cache size set to 1473832 entries Phase 2 - using internal log - zero log... zero_log: head block 1879314 tail block 1879310 ALERT: The filesystem has valuable metadata changes in a log which is being ignored because the -n option was used. Expect spurious inconsistencies which may be resolved by first mounting the filesystem to replay the log. - scan filesystem freespace and inode maps... sb_fdblocks 121671163, counted 123624614 - found root inode chunk Phase 3 - for each AG... - scan (but don't clear) agi unlinked lists... - process known inodes and perform inode discovery... - agno = 0 - agno = 1 - agno = 2 - agno = 3 - agno = 4 - agno = 5 - agno = 6 - agno = 7 - process newly discovered inodes... Phase 4 - check for duplicate blocks... - setting up duplicate extent list... - check for inodes claiming duplicate blocks... - agno = 0 - agno = 5 - agno = 7 - agno = 2 - agno = 4 - agno = 3 - agno = 6 - agno = 1 No modify flag set, skipping phase 5 Phase 6 - check inode connectivity... - traversing filesystem ... - agno = 0 - agno = 1 - agno = 2 - agno = 3 - agno = 4 - agno = 5 - agno = 6 - agno = 7 - traversal finished ... - moving disconnected inodes to lost+found ... Phase 7 - verify link counts... No modify flag set, skipping filesystem flush and exiting. XFS_REPAIR Summary Thu Jul 21 10:53:02 2022 Phase Start End Duration Phase 1: 07/21 10:51:57 07/21 10:51:57 Phase 2: 07/21 10:51:57 07/21 10:51:58 1 second Phase 3: 07/21 10:51:58 07/21 10:52:32 34 seconds Phase 4: 07/21 10:52:32 07/21 10:52:32 Phase 5: Skipped Phase 6: 07/21 10:52:32 07/21 10:53:02 30 seconds Phase 7: 07/21 10:53:02 07/21 10:53:02 Total run time: 1 minute, 5 seconds

Ah yes of course, silly me. Attached! Thanks! tower-diagnostics-20220721-0912.zip