FearlessUser

Awesome, I’ll take it. Much better than losing almost 24tb of shows and movies. I have them on disc, but the time it would take to restore that route is not ideal. Looked at several folders and didn’t see any naming issues yet, hopefully that’s the case for what’s left too. And now I really want to build a backup… Thank you

The "Fix" button was super quick, only took a few seconds for each drive. Checked the filesystem on both again and they said they were fine. Restarted the array in regular mode, and its doing a parity check. I see a bunch of my content is in some "lost+found" folders, am I good just moving them back to restore those files? Diagnostics added again if anyone wants to see them omega-diagnostics-20250808-1449.zip

Went to bed at a normal time last night and I’m off to work now so I’ll do that this afternoon when I get home, thanks

Rebuild finished. Stopped array and started in maintenance mode. Drive 1 says this when I check the filesystem status Phase 1 - find and verify superblock... Phase 2 - using internal log - zero log... - scan filesystem freespace and inode maps... - found root inode chunk Phase 3 - for each AG... - scan (but don't clear) agi unlinked lists... - process known inodes and perform inode discovery... - agno = 0 bad CRC for inode 128 bad CRC for inode 128, would rewrite would clear root inode 128 - agno = 1 - agno = 2 - agno = 3 - agno = 4 - agno = 5 - agno = 6 - agno = 7 - agno = 8 - agno = 9 - agno = 10 - process newly discovered inodes... Phase 4 - check for duplicate blocks... - setting up duplicate extent list... - check for inodes claiming duplicate blocks... - agno = 0 - agno = 1 - agno = 5 bad CRC for inode 128, would rewrite - agno = 4 would clear root inode 128 - agno = 2 - agno = 3 - agno = 6 - agno = 7 - agno = 8 - agno = 9 - agno = 10 No modify flag set, skipping phase 5 Phase 6 - check inode connectivity... would reinitialize root directory - traversing filesystem ... Metadata corruption detected at 0x488950, inode 0x80 dinode couldn't map inode 128, err = 117 - traversal finished ... - moving disconnected inodes to lost+found ... disconnected dir inode 131, would move to lost+found disconnected dir inode 132, would move to lost+found disconnected dir inode 12956713321, would move to lost+found Phase 7 - verify link counts... Metadata corruption detected at 0x488950, inode 0x80 dinode couldn't map inode 128, err = 117, can't compare link counts No modify flag set, skipping filesystem flush and exiting.Drive 2 says the same Phase 1 - find and verify superblock... Phase 2 - using internal log - zero log... - scan filesystem freespace and inode maps... - found root inode chunk Phase 3 - for each AG... - scan (but don't clear) agi unlinked lists... - process known inodes and perform inode discovery... - agno = 0 bad CRC for inode 128 bad CRC for inode 128, would rewrite Bad ctime nsec 1077710368 on inode 128, would reset to zero would clear root inode 128 - agno = 1 - agno = 2 - agno = 3 - agno = 4 - agno = 5 - agno = 6 - agno = 7 - agno = 8 - agno = 9 - agno = 10 - process newly discovered inodes... Phase 4 - check for duplicate blocks... - setting up duplicate extent list... - check for inodes claiming duplicate blocks... - agno = 0 - agno = 1 bad CRC for inode 128, would rewrite Bad ctime nsec 1077710368 on inode 128, would reset to zero would clear root inode 128 - agno = 2 - agno = 4 - agno = 6 - agno = 5 - agno = 3 - agno = 7 - agno = 8 - agno = 9 - agno = 10 No modify flag set, skipping phase 5 Phase 6 - check inode connectivity... would reinitialize root directory - traversing filesystem ... Metadata corruption detected at 0x488950, inode 0x80 dinode couldn't map inode 128, err = 117 - traversal finished ... - moving disconnected inodes to lost+found ... disconnected dir inode 131, would move to lost+found disconnected dir inode 191, would move to lost+found Phase 7 - verify link counts... Metadata corruption detected at 0x488950, inode 0x80 dinode couldn't map inode 128, err = 117, can't compare link counts No modify flag set, skipping filesystem flush and exiting.I'm left with a button on both that says "Fix" after it did the file system check on both drives, followed by "file system corruption detected". The link provided doesn't say what will happen if I hit that. I really just want to know if that's safe to do and will fix the issue, or if there is another route I should take. New diagnostics attached as well omega-diagnostics-20250807-1838.zip

Gotcha. It's around 50% with the rebuild when I woke up so I'll wait for that to finish and do the check later today

Sorry if this is a dumb question or if I'm just tired, but I added the drives to the server and immediately went to rebuild them, no preclear done per the response earlier this week. The new drive wouldn't have a file system to check, correct? And a format wouldn't clear any data on the disk because they're new drives, right? Apart from the 15% or so from the rebuild anyway.

1 and 2 were the ones that failed and were replaced. They've had a history of issues. Do it on the replacement drives, or the old drives?

New diagnostics, thank you omega-diagnostics-20250807-0023.zip

So I got the new drives today and plugged them in. Added the drives to the array and started it. Now it says it's working on a data rebuild, but the drives both say "unmountable, wrong or no filesystem". Is this normal until the rebuild completes (because they're fresh drives), or do I need to format them before adding them to the array?

Awesome that will save time. Thank you

I've had on and off again issues with 2 drives. Every time parity check would run, the drives would go offline with errors. It would take a few tries to rebuild, but they would eventually rebuild. Parity check ran again yesterday, and the drives failed again. Removed them from the array and added them back, and the rebuild was running up until about 80% or so. Both drives failed again. Now the dashboard shows them as emulated, but the Main tab shows them as "unmountable, wrong or no file system". I already purchased new drives today, they should show up Wednesday. Here are the diagnosticsomega-diagnostics-20250803-1455.zip I want to plan ahead for Wednesday and the next few days. I'll need to preclear the drives when I get them, then rebuild them. Is that where I'm at? I only use this for Plex. Some shows and movies won't play saying the drives aren't mounted. I'm hoping it's just because of the file system error, even though the dashboard shows them as being emulated. Is it going to be as simple as adding the new drives and rebuilding, or is there more work to do?

Rebuild took a few hours longer than estimated and completed overnight. I’m back in business. Thanks @JorgeB for the help on this, super appreciated!

Finally got around to doing this. When I started a new config, disk 1 was fine, but then disk 2 gave me 1061 errors and went offline shortly after starting the array up. These drives can't be failing in sequence like this, then be fine after adding them back to the array. I replaced these SAS to SATA cables I got from Amazon 6 years ago when I built the server. The parity drives were in different spots in the server, but these 2 data drives that were failing were on the same cable. Once it was replaced, I booted up and tried another new config, and so far so good. At this point the rebuild has been going for about 20 minutes, much longer than when the drives would almost immediately throw errors and fail. The rebuild says it should take around 30 hours, and all data drives are still showing active with no errors.. On the old cable, 3 of the 4 cables that split out to the drives were bent to a crease. Fingers crossed that's what caused the issues.

Geez I can’t believe I missed that. Long week. Thanks

2 question on this new config tool. Do I want to preserve any assignments? And how much should I worry about this message in the tool: "DO NOT USE THIS UTILITY THINKING IT WILL REBUILD A FAILED DRIVE - it will have the opposite effect of making it impossible to rebuild an existing failed drive - you have been warned!"