unmountable unsupported or no file system after rebuilding

[EH5]

I removed a disk that failed preclear and added another one to the system to preclear that.  when I started up the system one of the array drives was disabled and that data was emulated (i may not have the correct vocabulary for this).  I checked the forums and guessed that maybe i bumped the power or data cable for this drive so I reseated everything and started the array again.  when it booted it gave me the option of adding that disk in its old slot in the array and rebuilding to it so I did this.  After this started I realized the Dockers had not started up.  When the rebuild was underway another disk in the array seems to have failed.  When the rebuild finished I tried to get the dockers working again by doing a reboot of the system.  When the array started this time I got the "unmountable unsupported or no file system" error and the dockers failed to start.  Also I do not see any of the warnings about the data being emulated witch is concerning. 

tower-diagnostics-20240602-1333.zip

[JorgeB]

Check filesystem on disk6, run it without -n

[EH5]

Here is the bottom of the resulting report

 

<?>correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap correcting imap - agno = 1 - agno = 2 - agno = 3 - agno = 4 - agno = 5 - agno = 6 - agno = 7 - process newly discovered inodes... Phase 4 - check for duplicate blocks... - setting up duplicate extent list... - check for inodes claiming duplicate blocks... - agno = 0 - agno = 2 - agno = 3 - agno = 4 - agno = 5 - agno = 1 - agno = 7 - agno = 6 Phase 5 - rebuild AG headers and trees... - reset superblock... Phase 6 - check inode connectivity... - resetting contents of realtime bitmap and summary inodes - traversing filesystem ... - traversal finished ... - moving disconnected inodes to lost+found ... Phase 7 - verify and correct link counts... done

[EH5]

Here is the top of the report:

 

<?>Phase 1 - find and verify superblock... Phase 2 - using internal log - zero log... - scan filesystem freespace and inode maps... Metadata CRC error detected at 0x47191d, xfs_inobt block 0x18/0x1000 btree block 0/3 is suspect, error -74 root inode chunk not found Phase 3 - for each AG... - scan and clear agi unlinked lists... found inodes not in the inode allocation tree found inodes not in the inode allocation tree - process known inodes and perform inode discovery... - agno = 0 148c620e4680: Badness in key lookup (length) bp=(bno 0x1b800, len 4096 bytes) key=(bno 0x1b800, len 16384 bytes) 148c620e4680: Badness in key lookup (length) bp=(bno 0x1b820, len 4096 bytes) key=(bno 0x1b820, len 16384 bytes) 148c620e4680: Badness in key lookup (length) bp=(bno 0x491a80, len 4096 bytes) key=(bno 0x491a80, len 16384 bytes) 148c620e4680: Badness in key lookup (length) bp=(bno 0x491aa0, len 4096 bytes) key=(bno 0x491aa0, len 16384 bytes) 148c620e4680: Badness in key lookup (length) bp=(bno 0x8ae040, len 4096 bytes) key=(bno 0x8ae040, len 16384 bytes) 148c620e4680: Badness in key lookup (length) bp=(bno 0x8ae060, len 4096 bytes) key=(bno 0x8ae060, len 16384 bytes) 148c620e4680: Badness in key lookup (length) bp=(bno 0xf1d180, len 4096 bytes) key=(bno 0xf1d180, len 16384 bytes)<?>

[EH5]

I could not figure out the formatting so here is a text file which may be easier to read
 

Filesystemreport.txt

[JorgeB]

Re-start the array in normal mode, it should mount now, check contents and look for a lost+found folder.

[itimpi]

42 minutes ago, EH5 said:

I could not figure out the formatting

FYI:  The <> icon when entering a post allows you to supply text formatted correctly

[EH5]

Disk 6 seems to have been recovered but the Dockers have still not started.  Should I just follow the instructions I see on other docker failed to start posts?

 

Also, disk3 lists as not installed.  I could shut down the server and try to re-seat the power and data cables.  Is that a good idea?  I was able to click on the icon  next to disk 3 and see all the files so it seems like they are emulated even though there is no notice listing the drive as emulated.  Would it be reasonable to use unballanced to move everything off drive3 before doing anything else?

[JorgeB]

Post new diags.

 

 

[EH5]

Here is the new diagnostic file.  I have not rebooted the system since I got disk 6 back up

tower-diagnostics-20240604-0741.zip

[JorgeB]

Docker image is corrupt, delete and recreate:

https://docs.unraid.net/unraid-os/manual/docker-management/#re-create-the-docker-image-file

Also see below if you have any custom docker networks:

https://docs.unraid.net/unraid-os/manual/docker-management/#docker-custom-networks

 

Also check filesystem on disk6, run it without -n

 

Regarding disk3, I assume it was the unassigned WD serial ending in 366? If yes run an extended SMART test since it's showing some issues.

[EH5]

That got the dockers up and running. 

 

The missing disk is not 366.  The 366 disk is the one I installed to run preclear (it failed) so I think it is a bad disk.  I will run the extended smart test.

 

The missing disk is actually not showing up anywhere.  That is why I was considering shutting down and checking all the connections.  I was thinking either something is loose or it really failed hard. 

 

Just to confirm I should run check file system a second time on disk 6?  In the UI it looks ok right now.

 

Thanks for your help.

 

 

[JorgeB]

55 minutes ago, EH5 said:

The missing disk is actually not showing up anywhere.  That is why I was considering shutting down and checking all the connections.  I was thinking either something is loose or it really failed hard. 

You should do that.

 

56 minutes ago, EH5 said:

Just to confirm I should run check file system a second time on disk 6?  In the UI it looks ok right now.

If you didn't reboot yet and the disk is showing as mountable you can ignore that, I was likely seeing the old errors.