May 19, 20251 yr I had a disk that was marked offline while I was trying to get the system running with 3 new hbas. I have since migrated to a single hba and I brought the array back on with no issues but now I still have the one disk failed. I would just replace the disk and let it rebuild but I do not have faith in my parity drive because there was a parity check running when there was an issue with one of the hbas in the system. Anyone think its possible to repair the disk and get it back into the array? Thanks! This is the xfs_repair_status: Phase 1 - find and verify superblock... Phase 2 - using internal log - zero log... - scan filesystem freespace and inode maps... sb_fdblocks 1106282385, counted 1119907037 - found root inode chunk Phase 3 - for each AG... - scan (but don't clear) agi unlinked lists... - process known inodes and perform inode discovery... - agno = 0 bad CRC for inode 128 bad CRC for inode 128, would rewrite bad nextents 1124073472 for inode 128, would reset to 0 would clear root inode 128 - agno = 1 - agno = 2 - agno = 3 - agno = 4 - agno = 5 - agno = 6 - agno = 7 - process newly discovered inodes... Phase 4 - check for duplicate blocks... - setting up duplicate extent list... - check for inodes claiming duplicate blocks... - agno = 0 - agno = 2 - agno = 7 - agno = 3 - agno = 5 - agno = 1 - agno = 6 bad CRC for inode 128, would rewrite - agno = 4 bad nextents 1124073472 for inode 128, would reset to 0 would clear root inode 128 No modify flag set, skipping phase 5 Phase 6 - check inode connectivity... would reinitialize root directory - traversing filesystem ... Metadata corruption detected at 0x46f860, inode 0x80 dinode couldn't map inode 128, err = 117 - traversal finished ... - moving disconnected inodes to lost+found ... disconnected dir inode 131, would move to lost+found disconnected dir inode 36647070, would move to lost+found disconnected dir inode 2256759857, would move to lost+found disconnected dir inode 4319637530, would move to lost+found disconnected dir inode 4857193000, would move to lost+found disconnected dir inode 6454181966, would move to lost+found disconnected dir inode 6460265227, would move to lost+found disconnected dir inode 7048735315, would move to lost+found disconnected dir inode 8598394412, would move to lost+found disconnected dir inode 8637486748, would move to lost+found disconnected dir inode 8704629120, would move to lost+found disconnected dir inode 10760518383, would move to lost+found disconnected dir inode 11479625464, would move to lost+found disconnected dir inode 11643108476, would move to lost+found Phase 7 - verify link counts... Metadata corruption detected at 0x46f860, inode 0x80 dinode couldn't map inode 128, err = 117, can't compare link counts No modify flag set, skipping filesystem flush and exiting.
May 20, 20251 yr Author 1 hour ago, JorgeB said: Run xfs_repair again without -n Now what do I do? Thanks! Phase 1 - find and verify superblock... Phase 2 - using internal log - zero log... - scan filesystem freespace and inode maps... sb_fdblocks 1106282385, counted 1119907037 - found root inode chunk Phase 3 - for each AG... - scan and clear agi unlinked lists... - process known inodes and perform inode discovery... - agno = 0 bad CRC for inode 128 bad CRC for inode 128, will rewrite correcting nextents for inode 128, was 1124073472 - counted 0 cleared root inode 128 - agno = 1 - agno = 2 - agno = 3 - agno = 4 - agno = 5 - agno = 6 - agno = 7 - process newly discovered inodes... Phase 4 - check for duplicate blocks... - setting up duplicate extent list... - check for inodes claiming duplicate blocks... - agno = 0 - agno = 2 - agno = 4 - agno = 6 - agno = 5 - agno = 1 - agno = 3 - agno = 7 Phase 5 - rebuild AG headers and trees... - reset superblock... Phase 6 - check inode connectivity... reinitializing root directory - resetting contents of realtime bitmap and summary inodes - traversing filesystem ... - traversal finished ... - moving disconnected inodes to lost+found ... disconnected dir inode 131, moving to lost+found disconnected dir inode 36647070, moving to lost+found disconnected dir inode 2256759857, moving to lost+found disconnected dir inode 4319637530, moving to lost+found disconnected dir inode 4857193000, moving to lost+found disconnected dir inode 6454181966, moving to lost+found disconnected dir inode 6460265227, moving to lost+found disconnected dir inode 7048735315, moving to lost+found disconnected dir inode 8598394412, moving to lost+found disconnected dir inode 8637486748, moving to lost+found disconnected dir inode 8704629120, moving to lost+found disconnected dir inode 10760518383, moving to lost+found disconnected dir inode 11479625464, moving to lost+found disconnected dir inode 11643108476, moving to lost+found Phase 7 - verify and correct link counts... resetting inode 1260330676 nlinks from 2 to 16 done
May 20, 20251 yr Community Expert Now restart the array in normal mode and the emulated disk should mount, if the contents look OK you can then rebuild
May 21, 20251 yr Author 19 hours ago, JorgeB said:Now restart the array in normal mode and the emulated disk should mount, if the contents look OK you can then rebuildNow the problem I feel is that now under disk 1 it shows the proper disk where as before it did not. If I start the array it will start to rebuild over the disk and I will lose my chance to see if there is valid data on the disk that is not in the array. And yes I had one of the parity drives fail also.
May 21, 20251 yr Author 1 minute ago, JorgeB said:It won't start a rebuild, since it's showing red xThank you for your quick response!
May 21, 20251 yr Author When I start the array and click to browse the disk from the dashboard this is what I seeI would not have expected to see this on the emulated disk based on running the commands to "repair" the actual disk and the creation of the lost+found during that process.But I can see it does indeed say its emulated. This just confuses me a little bit. I assume I can mount sdf manually into another mount point so that I can see it. Thoughts?
May 21, 20251 yr Community Expert You can inspect the lost+found contents, most often there's not much there, but you can also try to mount the old disk with the UD plugin and check its contents, just stop the array and unassign the disk.
May 21, 20251 yr Author 7 minutes ago, JorgeB said:You can inspect the lost+found contents, most often there's not much there, but you can also try to mount the old disk with the UD plugin and check its contents, just stop the array and unassign the disk.When I mount the actual disk with UD it looks perfect. Exactly like what I would expect. I downloaded some files and spot checked them and they are fine. What would my process be at this point? I also have some reiserfs I actually want to remove from the array. I assume the answer is a "new config" where I assign the disks I want in the array along with the drives for parity and then just start the array. I dont want to mess something up so if there is a document that explains it that I can review. Does "new config" wipe out my installed plugins or docker images and things like that?Also it looks like the repair we did actually was working on the "emulated disk" configuration. Is that correct just so that I understand? That is the only thing which to me explains the lost+found folder present on the emulated disk.Thanks! Edited May 21, 20251 yr by dasoul
May 21, 20251 yr Community Expert 45 minutes ago, dasoul said:Also it looks like the repair we did actually was working on the "emulated disk" configuration. Is that correct just so that I understand?Correct.If the actual disk contents look better, instead of rebuilding you can do a new config to re-enable, but since there's a recent failed SMART test logged:SMART Extended Self-test Log Version: 1 (1 sectors)Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error# 1 Short offline Completed without error 00% 55534 -# 2 Short offline Completed: unknown failure 90% 55348 0I would recommend running an extended SMART test first, if the test passes, do a new config:-Tools -> New Config -> Retain current configuration: All -> ApplyYou will need to reassign old disk 1, then check parity is already valid and start the array, finally run a correcting parity check, since a few errors are expected.
May 22, 20251 yr Author On 5/21/2025 at 5:52 AM, JorgeB said:Correct.If the actual disk contents look better, instead of rebuilding you can do a new config to re-enable, but since there's a recent failed SMART test logged:SMART Extended Self-test Log Version: 1 (1 sectors)Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error# 1 Short offline Completed without error 00% 55534 -# 2 Short offline Completed: unknown failure 90% 55348 0I would recommend running an extended SMART test first, if the test passes, do a new config:-Tools -> New Config -> Retain current configuration: All -> ApplyYou will need to reassign old disk 1, then check parity is already valid and start the array, finally run a correcting parity check, since a few errors are expected.Thank you very much. I really appreciate it. Right now I have a few reiserfs that I am working to migrate the data to other disks via the unbalanced plugin. It will likely be another week before I am ready to do the new configuration and redo the data disks. On that note I also have new parity drives I would like to install. Do both parity disks actually need to be the same size? I was thinking I will replace parity 1 and then once I confirm the parity looks good then replace parity 2. I havent started using the higher capacity drives as data drives yet.
May 22, 20251 yr Community Expert 51 minutes ago, dasoul said:Do both parity disks actually need to be the same size?No. Parity drives can be different sizes as long as both of them are at least the size of the largest data drive.
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.