xfs_repair: fatal error -- couldn't map inode <>, err = 117

[phillipppp]

Had a power outage recently and had issues with Docker starting and killing all shares. After some looking around, I ran the `xfs_repair` on Disk 1 in my array and it reported issues. I re-ran without `-n` and cannot make it past the following:
 

...
Phase 5 - rebuild AG headers and trees...
        - reset superblock...
Phase 6 - check inode connectivity...
        - resetting contents of realtime bitmap and summary inodes
        - traversing filesystem ...
rebuilding directory inode 12955326179
Metadata corruption detected at 0x46fa05, inode 0x38983bd88 dinode

fatal error -- couldn't map inode 15192014216, err = 117

Starting the array outside maintenance mode now shows the disk with:
 

Unmountable: Unsupported or no file system

 

I am almost to the point of formatting the drive and rebuilding in place but want to check if there are any other options to try first.

Diagnostics attached.

tower-diagnostics-20231214-2011.zip

[JorgeB]

Try again, xfs_repair should never abort, it should always finish even if it would require some data loss, suggesting a xfs_repair bug.

[phillipppp]

Thanks @JorgeB . Unfortunately, I have tried multiple times all with the same result.  Would formatting the drive and doing a rebuild return the array to normal operation? 

Edited December 15, 2023 by phillipppp

[JorgeB]

Formatting the drive will delete all data and update parity, if there are no backups you can try a future release with newer xfsprogs in case that issue is fixed, or try something like UFS explorer to try and recover the data now.

[phillipppp]

Alright. I am on 6.12.6. Would there be any newer version? Should I replace the drive with a new one instead of formatting? 

[JorgeB]

4 minutes ago, phillipppp said:

Would there be any newer version?

Not at the moment.

 

5 minutes ago, phillipppp said:

Should I replace the drive with a new one instead of formatting?

You can try to see if the emulated disk is any different, most likely it won't, and you will need to disabled the disk to try that.

[phillipppp]

So, the only next steps are to wait for a newer version of unraid or restore the entire array from a backup? I was under the impression that, with my configuration, the array would be protected from loss of a single drive. I even keep a hot spare in the chassis for this. 
 

Are there no other options?

[trurl]

How are you running xfs repair? It is easy to get the command line wrong so be sure you run it from the webUI

[phillipppp]

I first ran from the web ui with `-n` and once it reported issues, I re-ran without the `-n`. I can provide the full output if it would help but it is quite large. 

[JorgeB]

1 hour ago, phillipppp said:

version of unraid or restore the entire array from a backup?

Not the entire array, that disk.

 

1 hour ago, phillipppp said:

I was under the impression that, with my configuration, the array would be protected from loss of a single drive. I even keep a hot spare in the chassis for this. 

Disk failure and filesystem corruption are different things, parity can protect from a disk failure.

[phillipppp]

2 minutes ago, JorgeB said:

Not the entire array, that disk.

Is it normal to keep disk-level backups and not data/share-based backups? 

[JorgeB]

I do because I use snapshots, and they can can only be disk by disk, but even if you have share backups you could use for example rsync and it would only restore the missing data.

[phillipppp]

I was able to get an image of the drive and mounted in an Ubuntu box booted on a USB. UFS was able to identify all the files on the drive in the same structure as my shares. 
 

What’s the best way to replace the disk without rebuilding the back without carrying the bad filesystem?

[trurl]

To get another disk into the array in place of that disk, you either have to rebuild onto the new disk, or New Config with the new disk in place of the old one, and rebuild parity instead.

 

Either way, after rebuild you can format the new disk in the array then copy data you recover from UFS to it.

[phillipppp]

Okay thanks, was curious if rebuilding could be avoided. Fingers crossed it goes smooth. 

[JorgeB]

If I understood correctly and the recovered data is outside the array, you can format that array disk and restore the data, no need to rebuild.

[phillipppp]

While still unresolved, I've got a thread going with the xfs distro. There may be more information there in case someone comes across a similar problem in the future and finds this topic.
https://lkml.kernel.org/linux-xfs/CACf8WVuHtDoJS1x5XzvBV99Z6Gp7bqbwDT4QsKsJrETtyyvFTg@mail.gmail.com/