Unmountable: No file system (Self inflicted problem!!) : RESOLVED

dja · December 15, 2023

Well, I'm on the naughty list.

I had an unmountable disk show up that parity rebuilt earlier this week. That went great.

This morning it went offline again.

I took everything down, cleaned, moved & checked cables etcetera. I did a new config because it wouldn't show online, and this worked for me once a while ago. That was bad. It didn't work. Of course, parity is being rewritten now, and the disk is showing the Unmountable: No file system message now.

Smart tests are okay, and the disk log seems fine. It ran the rebuild over 3 days without a problem, so I think the disk is actually ok. I just want to try to extract the data from it and re-add it to the array after a format. Unless I can somehow add it back in place?

Can someone help me un-eff this problem I created? Diags attached. TIA!!

zeus-diagnostics-20231215-1023.zip

Edited December 15, 2023 by dja
Resolved

itimpi · December 15, 2023

Have you tried following the procedure documented here in the online documentation accessible via the ‘Manual’ link at the bottom of the GUI or the DOCS link at the top of each forum page. The Unraid OS->Manual section in particular covers most features of the current Unraid release.

trurl · December 15, 2023

Note that none of the ways you worked on this problem earlier would fix unmountable.

Be sure to post the output of check filesystem so we can advise further.

dja · December 15, 2023

@trurl , Any harm in aborting the parity rebuild at this point?

Edited December 15, 2023 by dja

trurl · December 15, 2023

2 minutes ago, dja said:

@trurl , Any harm in aborting the parity rebuild at this point?

No harm that hasn't already been done.

dja · December 15, 2023

Well, here's what it is showing...

Phase 1 - find and verify superblock...
Phase 2 - using internal log
        - zero log...
ALERT: The filesystem has valuable metadata changes in a log which is being
ignored because the -n option was used.  Expect spurious inconsistencies
which may be resolved by first mounting the filesystem to replay the log.
        - scan filesystem freespace and inode maps...
Metadata CRC error detected at 0x44228d, xfs_bnobt block 0x9e032a8/0x1000
btree block 0/20710997 is suspect, error -74
bad magic # 0x80059531 in btbno block 0/20710997
Metadata CRC error detected at 0x44228d, xfs_cntbt block 0x9e1a340/0x1000
btree block 0/20722792 is suspect, error -74
bad magic # 0x8005952b in btcnt block 0/20722792
Metadata CRC error detected at 0x47494d, xfs_refcountbt block 0x9da1a38/0x1000
btree block 0/20661063 is suspect, error -74
bad magic # 0x8005956c in refcount btree block 0/20661063
bad refcountbt block count 0, saw 1
agf_freeblks 1871579, counted 1859078 in ag 0
agf_btreeblks 4, counted 1 in ag 0
sb_fdblocks 597112460, counted 614127692
        - found root inode chunk
Phase 3 - for each AG...
        - scan (but don't clear) agi unlinked lists...
        - process known inodes and perform inode discovery...
        - agno = 0
        - agno = 1
        - agno = 2
        - agno = 3
        - agno = 4
        - agno = 5
        - agno = 6
        - agno = 7
        - agno = 8
        - agno = 9
        - process newly discovered inodes...
Phase 4 - check for duplicate blocks...
        - setting up duplicate extent list...
free space (0,20656014-20656191) only seen by one free space btree
free space (0,20656281-20656439) only seen by one free space btree
free space (0,20656486-20658631) only seen by one free space btree
free space (0,20666871-20667933) only seen by one free space btree
free space (0,20726243-20728973) only seen by one free space btree
free space (0,20734026-20734422) only seen by one free space btree
free space (0,20742854-20743996) only seen by one free space btree
free space (0,20747810-20748474) only seen by one free space btree
free space (0,20768918-20769318) only seen by one free space btree
free space (0,20836538-20838162) only seen by one free space btree
free space (0,28455444-28456305) only seen by one free space btree
free space (0,28462145-28463735) only seen by one free space btree
free space (0,28473001-28473983) only seen by one free space btree
free space (0,28480531-28481802) only seen by one free space btree
free space (0,38999206-38999966) only seen by one free space btree
free space (0,39014320-39016845) only seen by one free space btree
free space (0,39029946-39031652) only seen by one free space btree
free space (0,39034226-39036786) only seen by one free space btree
free space (0,39039918-39040327) only seen by one free space btree
free space (0,39041880-39042091) only seen by one free space btree
free space (0,39062643-39062868) only seen by one free space btree
free space (0,39070660-39071928) only seen by one free space btree
free space (0,39077119-39079851) only seen by one free space btree
free space (0,39084940-39085092) only seen by one free space btree
free space (0,39088431-39089146) only seen by one free space btree
free space (0,39124958-39125468) only seen by one free space btree
free space (0,39158273-39159725) only seen by one free space btree
free space (0,39161516-39162187) only seen by one free space btree
free space (0,39165241-39166338) only seen by one free space btree
free space (0,39170328-39170579) only seen by one free space btree
free space (0,39189590-39190801) only seen by one free space btree
free space (0,39197354-39198619) only seen by one free space btree
free space (0,39205178-39206531) only seen by one free space btree
free space (0,39220082-39220801) only seen by one free space btree
free space (0,39224063-39224212) only seen by one free space btree
free space (0,42294313-42295130) only seen by one free space btree
free space (0,42313334-42316005) only seen by one free space btree
free space (0,42332210-42333178) only seen by one free space btree
free space (0,42335577-42335777) only seen by one free space btree
free space (0,42345143-42350003) only seen by one free space btree
free space (0,42360763-42361141) only seen by one free space btree
free space (0,42362036-42362205) only seen by one free space btree
free space (0,42362921-42365572) only seen by one free space btree
free space (0,42373303-42373443) only seen by one free space btree
free space (0,42379609-42379758) only seen by one free space btree
free space (0,42380191-42381751) only seen by one free space btree
free space (0,42383092-42384455) only seen by one free space btree
free space (0,53143601-53146239) only seen by one free space btree
free space (0,53162192-53163507) only seen by one free space btree
free space (0,53200670-53203347) only seen by one free space btree
free space (0,53213039-53213574) only seen by one free space btree
free space (0,53216020-53216189) only seen by one free space btree
free space (0,59587924-59588070) only seen by one free space btree
free space (0,59603391-59603585) only seen by one free space btree
free space (0,59612370-59614741) only seen by one free space btree
free space (0,59617814-59620365) only seen by one free space btree
free space (0,69466076-69466268) only seen by one free space btree
free space (0,69474524-69474679) only seen by one free space btree
free space (0,69476815-69479454) only seen by one free space btree
free space (0,69481187-69481834) only seen by one free space btree
free space (0,69488658-69488836) only seen by one free space btree
free space (0,69505007-69505148) only seen by one free space btree
free space (0,69517863-69518186) only seen by one free space btree
free space (0,69530148-69530958) only seen by one free space btree
free space (0,69541269-69542572) only seen by one free space btree
free space (0,69543148-69545783) only seen by one free space btree
free space (0,69558010-69559423) only seen by one free space btree
free space (0,69572220-69574833) only seen by one free space btree
free space (0,69583632-69584935) only seen by one free space btree
free space (0,69586512-69586856) only seen by one free space btree
free space (0,69612840-69614142) only seen by one free space btree
free space (0,69615734-69616331) only seen by one free space btree
free space (0,69620727-69622076) only seen by one free space btree
free space (0,69637345-69638681) only seen by one free space btree
free space (0,81551137-81553762) only seen by one free space btree
free space (0,81571368-81572836) only seen by one free space btree
free space (0,81578494-81582580) only seen by one free space btree
free space (0,88611369-88611821) only seen by one free space btree
free space (0,88618870-88619366) only seen by one free space btree
free space (0,101675703-101678235) only seen by one free space btree
free space (0,117445720-117446633) only seen by one free space btree
free space (0,117448849-117450193) only seen by one free space btree
free space (0,121305721-121308185) only seen by one free space btree
free space (0,121312315-121312492) only seen by one free space btree
free space (0,121322874-121325607) only seen by one free space btree
free space (0,121339483-121339929) only seen by one free space btree
free space (0,121346799-121353451) only seen by one free space btree
free space (0,129119850-129120016) only seen by one free space btree
free space (0,129122783-129124169) only seen by one free space btree
free space (0,129129103-129129643) only seen by one free space btree
free space (0,129133434-129135014) only seen by one free space btree
free space (0,129161979-129162235) only seen by one free space btree
free space (0,129166164-129168728) only seen by one free space btree
free space (0,129179377-129180689) only seen by one free space btree
free space (0,129183378-129183555) only seen by one free space btree
free space (0,129198228-129198482) only seen by one free space btree
free space (0,129199883-129202436) only seen by one free space btree
free space (0,129215779-129218721) only seen by one free space btree
free space (0,129225363-129226696) only seen by one free space btree
free space (0,129240283-129243016) only seen by one free space btree
free space (0,129257481-129258307) only seen by one free space btree
free space (0,129260079-129261245) only seen by one free space btree
free space (0,129294788-129299286) only seen by one free space btree
free space (0,129310891-129311863) only seen by one free space btree
free space (0,129316272-129316411) only seen by one free space btree
free space (0,131199472-131200962) only seen by one free space btree
free space (0,131207965-131208220) only seen by one free space btree
free space (0,131223004-131225682) only seen by one free space btree
free space (0,131253426-131256071) only seen by one free space btree
free space (0,131273970-131275714) only seen by one free space btree
free space (0,131277061-131277355) only seen by one free space btree
free space (0,131334947-131337489) only seen by one free space btree
free space (0,131356019-131356265) only seen by one free space btree
free space (0,144750428-144752246) only seen by one free space btree
free space (0,144780119-144781690) only seen by one free space btree
free space (0,144785734-144788394) only seen by one free space btree
free space (0,144789398-144789941) only seen by one free space btree
free space (0,156299257-156304610) only seen by one free space btree
free space (0,156306817-156308302) only seen by one free space btree
free space (0,156333695-156334164) only seen by one free space btree
free space (0,156336791-156337474) only seen by one free space btree
free space (0,156340925-156341120) only seen by one free space btree
free space (0,156346907-156347485) only seen by one free space btree
free space (0,156348871-156351783) only seen by one free space btree
free space (0,156356179-156357527) only seen by one free space btree
free space (0,156362637-156363609) only seen by one free space btree
free space (0,156374161-156374336) only seen by one free space btree
free space (0,156378195-156379978) only seen by one free space btree
free space (0,156381741-156382429) only seen by one free space btree
free space (0,156410274-156412884) only seen by one free space btree
free space (0,156430456-156431126) only seen by one free space btree
free space (0,156432366-156432644) only seen by one free space btree
free space (0,156457361-156457535) only seen by one free space btree
free space (0,156461213-156461644) only seen by one free space btree
free space (0,156467785-156477582) only seen by one free space btree
free space (0,156481033-156482563) only seen by one free space btree
free space (0,156484658-156485521) only seen by one free space btree
free space (0,156501080-156501261) only seen by one free space btree
free space (0,156505785-156508396) only seen by one free space btree
free space (0,156509379-156510794) only seen by one free space btree
free space (0,156512963-156513101) only seen by one free space btree
free space (0,156516552-156519146) only seen by one free space btree
free space (0,156525029-156527375) only seen by one free space btree
free space (0,156532474-156533951) only seen by one free space btree
free space (0,156535346-156538229) only seen by one free space btree
free space (0,156546015-156549239) only seen by one free space btree
free space (0,156556165-156556427) only seen by one free space btree
free space (0,156560678-156561512) only seen by one free space btree
free space (0,156563928-156564473) only seen by one free space btree
free space (0,156569395-156570053) only seen by one free space btree
free space (0,156576909-156577178) only seen by one free space btree
free space (0,156584774-156585400) only seen by one free space btree
free space (0,156588851-156590148) only seen by one free space btree
free space (0,156595579-156596105) only seen by one free space btree
free space (0,156596145-156596290) only seen by one free space btree
free space (0,156596458-156596793) only seen by one free space btree
free space (0,160471966-160474640) only seen by one free space btree
free space (0,160478573-160479137) only seen by one free space btree
free space (0,160491220-160491372) only seen by one free space btree
free space (0,160493702-160494247) only seen by one free space btree
free space (0,160495640-160496930) only seen by one free space btree
free space (0,160510713-160511132) only seen by one free space btree
free space (0,160515599-160516441) only seen by one free space btree
free space (0,160519892-160520514) only seen by one free space btree
free space (0,160521815-160522977) only seen by one free space btree
free space (0,160526631-160529452) only seen by one free space btree
free space (0,160529822-160530093) only seen by one free space btree
free space (0,182462344-182463007) only seen by one free space btree
free space (0,182469795-182471152) only seen by one free space btree
free space (0,182475097-182475355) only seen by one free space btree
free space (0,182478806-182480945) only seen by one free space btree
free space (0,182487838-182488129) only seen by one free space btree
free space (0,182491114-182494084) only seen by one free space btree
free space (0,182499051-182499793) only seen by one free space btree
free space (0,182504033-182504425) only seen by one free space btree
free space (0,182508420-182511348) only seen by one free space btree
free space (0,182514370-182515018) only seen by one free space btree
free space (0,182518336-182521189) only seen by one free space btree
free space (0,182522760-182525870) only seen by one free space btree
free space (0,182534388-182536391) only seen by one free space btree
free space (0,182546440-182546688) only seen by one free space btree
free space (0,182547120-182547577) only seen by one free space btree
free space (0,182547715-182548099) only seen by one free space btree
free space (0,185319314-185320155) only seen by one free space btree
free space (0,185321433-185321627) only seen by one free space btree
free space (0,185330425-185331243) only seen by one free space btree
free space (0,185341491-185342344) only seen by one free space btree
free space (0,185356079-185356811) only seen by one free space btree
free space (0,185362800-185365701) only seen by one free space btree
free space (0,185371920-185372849) only seen by one free space btree
free space (0,185375323-185376005) only seen by one free space btree
free space (0,185381844-185382680) only seen by one free space btree
free space (0,185383427-185384710) only seen by one free space btree
free space (0,185387665-185388396) only seen by one free space btree
free space (0,185391620-185394642) only seen by one free space btree
free space (0,185398093-185398326) only seen by one free space btree
free space (0,185398430-185399764) only seen by one free space btree
free space (0,185406848-185407884) only seen by one free space btree
free space (0,185414782-185415174) only seen by one free space btree
free space (0,185420388-185421313) only seen by one free space btree
free space (0,185425486-185427843) only seen by one free space btree
free space (0,185430641-185430890) only seen by one free space btree
free space (0,185431315-185431683) only seen by one free space btree
free space (0,190479297-190479437) only seen by one free space btree
free space (0,190482544-190483194) only seen by one free space btree
free space (0,190485540-190486241) only seen by one free space btree
free space (0,197406369-197407340) only seen by one free space btree
free space (0,197412226-197412764) only seen by one free space btree
free space (0,197418061-197420400) only seen by one free space btree
free space (0,197424909-197426718) only seen by one free space btree
free space (0,197439040-197439401) only seen by one free space btree
free space (0,197447796-197448566) only seen by one free space btree
free space (0,197460642-197462335) only seen by one free space btree
free space (0,197467345-197467794) only seen by one free space btree
free space (0,197475681-197476670) only seen by one free space btree
free space (0,197483853-197484303) only seen by one free space btree
free space (0,197505332-197505476) only seen by one free space btree
free space (0,197515464-197515696) only seen by one free space btree
free space (0,197520643-197521408) only seen by one free space btree
free space (0,197531318-197532460) only seen by one free space btree
free space (0,197538190-197540390) only seen by one free space btree
free space (0,197551728-197553315) only seen by one free space btree
free space (0,197555620-197556711) only seen by one free space btree
free space (0,197557113-197557758) only seen by one free space btree
free space (0,197560112-197560784) only seen by one free space btree
free space (0,197563823-197564438) only seen by one free space btree
free space (0,197572374-197574403) only seen by one free space btree
free space (0,197579083-197579654) only seen by one free space btree
free space (0,197585369-197585585) only seen by one free space btree
free space (0,197588523-197588714) only seen by one free space btree
free space (0,197598743-197599693) only seen by one free space btree
free space (0,197607950-197608480) only seen by one free space btree
free space (0,197613220-197614279) only seen by one free space btree
free space (0,197619011-197619421) only seen by one free space btree
free space (0,197622095-197622627) only seen by one free space btree
free space (0,197651183-197652514) only seen by one free space btree
free space (0,197655052-197655369) only seen by one free space btree
free space (0,197657083-197657628) only seen by one free space btree
free space (0,202301146-202301622) only seen by one free space btree
free space (0,202313742-202314632) only seen by one free space btree
free space (0,202318083-202318224) only seen by one free space btree
free space (0,202322112-202322423) only seen by one free space btree
free space (0,202335414-202335677) only seen by one free space btree
free space (0,202342507-202343095) only seen by one free space btree
free space (0,202348358-202348845) only seen by one free space btree
free space (0,209380948-209381152) only seen by one free space btree
free space (0,209383505-209384069) only seen by one free space btree
free space (0,209393292-209395298) only seen by one free space btree
free space (0,209400955-209401905) only seen by one free space btree
free space (0,209403089-209403408) only seen by one free space btree
free space (0,209410725-209412566) only seen by one free space btree
free space (0,209412677-209413314) only seen by one free space btree
free space (0,209418634-209419596) only seen by one free space btree
free space (0,209426521-209426964) only seen by one free space btree
free space (0,216487807-216487973) only seen by one free space btree
free space (0,216496192-216496772) only seen by one free space btree
free space (0,216514150-216515419) only seen by one free space btree
free space (0,216521028-216522628) only seen by one free space btree
free space (0,216527429-216528124) only seen by one free space btree
free space (0,216558771-216559030) only seen by one free space btree
free space (0,216564341-216564935) only seen by one free space btree
free space (0,216567854-216569450) only seen by one free space btree
free space (0,216581721-216582221) only seen by one free space btree
free space (0,216593282-216593714) only seen by one free space btree
free space (0,216600966-216602693) only seen by one free space btree
free space (0,216602800-216603122) only seen by one free space btree
free space (0,216605491-216605953) only seen by one free space btree
free space (0,217588822-217589062) only seen by one free space btree
free space (0,217606958-217607133) only seen by one free space btree
free space (0,217614916-217615397) only seen by one free space btree
free space (0,217617628-217617774) only seen by one free space btree
free space (0,218296906-218297380) only seen by one free space btree
free space (0,218299175-218299681) only seen by one free space btree
free space (0,218301220-218429111) only seen by one free space btree
free space (0,219359413-219403492) only seen by one free space btree
free space (0,219472150-219477671) only seen by one free space btree
free space (0,222647386-222662496) only seen by one free space btree
free space (0,230326824-231142677) only seen by one free space btree
free space (0,235168426-235193355) only seen by one free space btree
free space (0,244336938-244337378) only seen by one free space btree
free space (0,249694404-249728343) only seen by one free space btree
free space (0,258339135-258539642) only seen by one free space btree
free space (0,259588203-259655560) only seen by one free space btree
free space (0,261944853-261997481) only seen by one free space btree
free space (0,268281952-268435454) only seen by one free space btree
        - check for inodes claiming duplicate blocks...
        - agno = 0
        - agno = 1
        - agno = 9
        - agno = 2
        - agno = 4
        - agno = 3
        - agno = 5
        - agno = 6
        - agno = 7
        - agno = 8
would rebuild corrupt refcount btrees.
No modify flag set, skipping phase 5
Phase 6 - check inode connectivity...
        - traversing filesystem ...
        - traversal finished ...
        - moving disconnected inodes to lost+found ...
Phase 7 - verify link counts...
Maximum metadata LSN (1249326922:1882287980) is ahead of log (1:4103733).
Would format log to cycle 1249326925.
No modify flag set, skipping filesystem flush and exiting.

JorgeB · December 15, 2023

Run it again without -n or nothing will be done.

dja · December 15, 2023

8 minutes ago, JorgeB said:

Run it again without -n or nothing will be done.

Getting this in maint. mode when doing that.


Phase 1 - find and verify superblock...
Phase 2 - using internal log
        - zero log...
ERROR: The filesystem has valuable metadata changes in a log which needs to
be replayed.  Mount the filesystem to replay the log, and unmount it before
re-running xfs_repair.  If you are unable to mount the filesystem, then use
the -L option to destroy the log and attempt a repair.
Note that destroying the log may cause corruption -- please attempt a mount
of the filesystem before doing this.

Edited December 15, 2023 by dja

JorgeB · December 15, 2023

Use -L

dja · December 15, 2023

Ok. I'm afraid to do anything at this point lol...what next? Thanks for the help.


Phase 1 - find and verify superblock...
Phase 2 - using internal log
        - zero log...
ALERT: The filesystem has valuable metadata changes in a log which is being
destroyed because the -L option was used.
        - scan filesystem freespace and inode maps...
clearing needsrepair flag and regenerating metadata
Metadata CRC error detected at 0x44228d, xfs_bnobt block 0x9e032a8/0x1000
btree block 0/20710997 is suspect, error -74
bad magic # 0x80059531 in btbno block 0/20710997
Metadata CRC error detected at 0x44228d, xfs_cntbt block 0x9e1a340/0x1000
btree block 0/20722792 is suspect, error -74
bad magic # 0x8005952b in btcnt block 0/20722792
Metadata CRC error detected at 0x47494d, xfs_refcountbt block 0x9da1a38/0x1000
btree block 0/20661063 is suspect, error -74
bad magic # 0x8005956c in refcount btree block 0/20661063
bad refcountbt block count 0, saw 1
agf_freeblks 1871579, counted 1859078 in ag 0
agf_btreeblks 4, counted 1 in ag 0
sb_fdblocks 597112460, counted 614127692
        - found root inode chunk
Phase 3 - for each AG...
        - scan and clear agi unlinked lists...
        - process known inodes and perform inode discovery...
        - agno = 0
        - agno = 1
        - agno = 2
        - agno = 3
        - agno = 4
        - agno = 5
        - agno = 6
        - agno = 7
        - agno = 8
        - agno = 9
        - process newly discovered inodes...
Phase 4 - check for duplicate blocks...
        - setting up duplicate extent list...
        - check for inodes claiming duplicate blocks...
        - agno = 0
        - agno = 2
        - agno = 5
        - agno = 8
        - agno = 6
        - agno = 4
        - agno = 7
        - agno = 1
        - agno = 3
        - agno = 9
Phase 5 - rebuild AG headers and trees...
        - reset superblock...
Phase 6 - check inode connectivity...
        - resetting contents of realtime bitmap and summary inodes
        - traversing filesystem ...
        - traversal finished ...
        - moving disconnected inodes to lost+found ...
Phase 7 - verify and correct link counts...
Maximum metadata LSN (1249326922:1882287980) is ahead of log (1:2).
Format log to cycle 1249326925.
done

itimpi · December 15, 2023

Restart the array in normal mode.

JorgeB · December 15, 2023

It should be fixed, start in normal mode, look for a lost+found folder.

dja · December 15, 2023

Thanks guys- THAT WORKED!!!

Appears everything survived, just a couple lost and founds.

Unmountable: No file system (Self inflicted problem!!) : RESOLVED

Recommended Posts

dja

Link to comment

itimpi

Link to comment

trurl

Link to comment

dja

Link to comment

trurl

Link to comment

dja

Link to comment

JorgeB

Link to comment

dja

Link to comment

JorgeB

Link to comment

dja

Link to comment

itimpi

Link to comment

JorgeB

Link to comment

dja

Link to comment

Join the conversation