Unassigned Drive not mounting after power outage.

[ManBullMoose]

Hello everyone.

I'm an amateur with this system yet, and need some help. I have a system where I have unnassigned drives storing my main gaming vm. This m.2 drive is in unassigned devices. We had a bad winter storm here in Michigan, and our power went out. (I don't have a ups yet, hoping Santa leaves me one) Now this particular drive (drive 3) refuses to mount. I farted around on here and found how to use xfs check on drives while in maintenance mode. As far as I can tell, I can only do this option for disks on the array, the prompts do not appear on the drives check marks while in the unassigned devices tab. I thought I could move the drive into the array side, but it will say that it will erase everything on the disk once the array was started. 

This isn't the main disk of the vm, just another game storage drive, but I'd like to not wipe the whole thing. How does one perform xfs repair on an unassigned device?

These are the logs:

Dec 23 17:31:53 Tempest kernel: nvme0n1: p1
Dec 23 17:32:27 Tempest  emhttpd: Samsung_SSD_980_PRO_2TB_S6B0NL0T618459Z (nvme0n1) 512 3907029168
Dec 23 17:32:28 Tempest  emhttpd: read SMART /dev/nvme0n1
Dec 23 17:33:16 Tempest unassigned.devices: Adding partition 'nvme0n1p1'...
Dec 23 17:33:16 Tempest unassigned.devices: Mounting partition 'nvme0n1p1' at mountpoint '/mnt/disks/S6B0NL0T618459Z'...
Dec 23 17:33:16 Tempest unassigned.devices: Mount drive command: /sbin/mount -t 'xfs' -o rw,noatime,nodiratime,discard '/dev/nvme0n1p1' '/mnt/disks/S6B0NL0T618459Z'
Dec 23 17:33:16 Tempest kernel: XFS (nvme0n1p1): Mounting V5 Filesystem
Dec 23 17:33:16 Tempest kernel: XFS (nvme0n1p1): Starting recovery (logdev: internal)
Dec 23 17:33:16 Tempest kernel: XFS (nvme0n1p1): Metadata CRC error detected at xfs_refcountbt_read_verify+0x12/0x5a [xfs], xfs_refcountbt block 0x3a382058 
Dec 23 17:33:16 Tempest kernel: XFS (nvme0n1p1): Unmount and run xfs_repair
Dec 23 17:33:16 Tempest kernel: XFS (nvme0n1p1): First 128 bytes of corrupted metadata buffer:
Dec 23 17:33:16 Tempest kernel: XFS (nvme0n1p1): metadata I/O error in "xfs_btree_read_buf_block.constprop.0+0x7a/0xc7 [xfs]" at daddr 0x3a382058 len 8 error 74
Dec 23 17:33:16 Tempest kernel: XFS (nvme0n1p1): Failed to recover leftover CoW staging extents, err -117.
Dec 23 17:33:16 Tempest kernel: XFS (nvme0n1p1): Filesystem has been shut down due to log error (0x2).
Dec 23 17:33:16 Tempest kernel: XFS (nvme0n1p1): Please unmount the filesystem and rectify the problem(s).
Dec 23 17:33:16 Tempest kernel: XFS (nvme0n1p1): Ending recovery (logdev: internal)
Dec 23 17:33:16 Tempest kernel: XFS (nvme0n1p1): Error -5 reserving per-AG metadata reserve pool.
Dec 23 17:33:16 Tempest unassigned.devices: Mount of 'nvme0n1p1' failed: 'mount: /mnt/disks/S6B0NL0T618459Z: can't read superblock on /dev/nvme0n1p1.        dmesg(1) may have more information after failed mount system call. '
Dec 23 17:33:43 Tempest unassigned.devices: Adding partition 'nvme0n1p1'...
Dec 23 17:33:43 Tempest unassigned.devices: Mounting partition 'nvme0n1p1' at mountpoint '/mnt/disks/S6B0NL0T618459Z'...
Dec 23 17:33:43 Tempest unassigned.devices: Mount drive command: /sbin/mount -t 'xfs' -o rw,noatime,nodiratime,discard '/dev/nvme0n1p1' '/mnt/disks/S6B0NL0T618459Z'
Dec 23 17:33:43 Tempest kernel: XFS (nvme0n1p1): Mounting V5 Filesystem
Dec 23 17:33:43 Tempest kernel: XFS (nvme0n1p1): Starting recovery (logdev: internal)
Dec 23 17:33:43 Tempest kernel: XFS (nvme0n1p1): Metadata CRC error detected at xfs_refcountbt_read_verify+0x12/0x5a [xfs], xfs_refcountbt block 0x3a382058 
Dec 23 17:33:43 Tempest kernel: XFS (nvme0n1p1): Unmount and run xfs_repair
Dec 23 17:33:43 Tempest kernel: XFS (nvme0n1p1): First 128 bytes of corrupted metadata buffer:
Dec 23 17:33:43 Tempest kernel: XFS (nvme0n1p1): metadata I/O error in "xfs_btree_read_buf_block.constprop.0+0x7a/0xc7 [xfs]" at daddr 0x3a382058 len 8 error 74
Dec 23 17:33:43 Tempest kernel: XFS (nvme0n1p1): Failed to recover leftover CoW staging extents, err -117.
Dec 23 17:33:43 Tempest kernel: XFS (nvme0n1p1): Filesystem has been shut down due to log error (0x2).
Dec 23 17:33:43 Tempest kernel: XFS (nvme0n1p1): Please unmount the filesystem and rectify the problem(s).
Dec 23 17:33:43 Tempest kernel: XFS (nvme0n1p1): Ending recovery (logdev: internal)
Dec 23 17:33:43 Tempest kernel: XFS (nvme0n1p1): Error -5 reserving per-AG metadata reserve pool.
Dec 23 17:33:43 Tempest unassigned.devices: Mount of 'nvme0n1p1' failed: 'mount: /mnt/disks/S6B0NL0T618459Z: can't read superblock on /dev/nvme0n1p1.        dmesg(1) may have more information after failed mount system call. '
Dec 23 17:34:33 Tempest unassigned.devices: Adding partition 'nvme0n1p1'...
Dec 23 17:34:33 Tempest unassigned.devices: Mounting partition 'nvme0n1p1' at mountpoint '/mnt/disks/S6B0NL0T618459Z'...
Dec 23 17:34:33 Tempest unassigned.devices: Mount drive command: /sbin/mount -t 'xfs' -o rw,noatime,nodiratime,discard '/dev/nvme0n1p1' '/mnt/disks/S6B0NL0T618459Z'
Dec 23 17:34:33 Tempest kernel: XFS (nvme0n1p1): Mounting V5 Filesystem
Dec 23 17:34:33 Tempest kernel: XFS (nvme0n1p1): Starting recovery (logdev: internal)
Dec 23 17:34:33 Tempest kernel: XFS (nvme0n1p1): Metadata CRC error detected at xfs_refcountbt_read_verify+0x12/0x5a [xfs], xfs_refcountbt block 0x3a382058 
Dec 23 17:34:33 Tempest kernel: XFS (nvme0n1p1): Unmount and run xfs_repair
Dec 23 17:34:33 Tempest kernel: XFS (nvme0n1p1): First 128 bytes of corrupted metadata buffer:
Dec 23 17:34:33 Tempest kernel: XFS (nvme0n1p1): metadata I/O error in "xfs_btree_read_buf_block.constprop.0+0x7a/0xc7 [xfs]" at daddr 0x3a382058 len 8 error 74
Dec 23 17:34:33 Tempest kernel: XFS (nvme0n1p1): Failed to recover leftover CoW staging extents, err -117.
Dec 23 17:34:33 Tempest kernel: XFS (nvme0n1p1): Filesystem has been shut down due to log error (0x2).
Dec 23 17:34:33 Tempest kernel: XFS (nvme0n1p1): Please unmount the filesystem and rectify the problem(s).
Dec 23 17:34:33 Tempest kernel: XFS (nvme0n1p1): Ending recovery (logdev: internal)
Dec 23 17:34:33 Tempest kernel: XFS (nvme0n1p1): Error -5 reserving per-AG metadata reserve pool.
Dec 23 17:34:33 Tempest unassigned.devices: Mount of 'nvme0n1p1' failed: 'mount: /mnt/disks/S6B0NL0T618459Z: can't read superblock on /dev/nvme0n1p1.        dmesg(1) may have more information after failed mount system call. '
Dec 23 17:34:50 Tempest unassigned.devices: Adding partition 'nvme0n1p1'...
Dec 23 17:34:50 Tempest unassigned.devices: Mounting partition 'nvme0n1p1' at mountpoint '/mnt/disks/S6B0NL0T618459Z'...
Dec 23 17:34:50 Tempest unassigned.devices: Mount drive command: /sbin/mount -t 'xfs' -o rw,noatime,nodiratime,discard '/dev/nvme0n1p1' '/mnt/disks/S6B0NL0T618459Z'
Dec 23 17:34:50 Tempest kernel: XFS (nvme0n1p1): Mounting V5 Filesystem
Dec 23 17:34:50 Tempest kernel: XFS (nvme0n1p1): Starting recovery (logdev: internal)
Dec 23 17:34:50 Tempest kernel: XFS (nvme0n1p1): Metadata CRC error detected at xfs_refcountbt_read_verify+0x12/0x5a [xfs], xfs_refcountbt block 0x3a382058 
Dec 23 17:34:50 Tempest kernel: XFS (nvme0n1p1): Unmount and run xfs_repair
Dec 23 17:34:50 Tempest kernel: XFS (nvme0n1p1): First 128 bytes of corrupted metadata buffer:
Dec 23 17:34:50 Tempest kernel: XFS (nvme0n1p1): metadata I/O error in "xfs_btree_read_buf_block.constprop.0+0x7a/0xc7 [xfs]" at daddr 0x3a382058 len 8 error 74
Dec 23 17:34:50 Tempest kernel: XFS (nvme0n1p1): Failed to recover leftover CoW staging extents, err -117.
Dec 23 17:34:50 Tempest kernel: XFS (nvme0n1p1): Filesystem has been shut down due to log error (0x2).
Dec 23 17:34:50 Tempest kernel: XFS (nvme0n1p1): Please unmount the filesystem and rectify the problem(s).
Dec 23 17:34:50 Tempest kernel: XFS (nvme0n1p1): Ending recovery (logdev: internal)
Dec 23 17:34:50 Tempest kernel: XFS (nvme0n1p1): Error -5 reserving per-AG metadata reserve pool.
Dec 23 17:34:50 Tempest unassigned.devices: Mount of 'nvme0n1p1' failed: 'mount: /mnt/disks/S6B0NL0T618459Z: can't read superblock on /dev/nvme0n1p1.        dmesg(1) may have more information after failed mount system call. '
Dec 23 17:35:50 Tempest unassigned.devices: Adding partition 'nvme0n1p1'...
Dec 23 17:35:50 Tempest unassigned.devices: Mounting partition 'nvme0n1p1' at mountpoint '/mnt/disks/S6B0NL0T618459Z'...
Dec 23 17:35:50 Tempest unassigned.devices: Mount drive command: /sbin/mount -t 'xfs' -o rw,noatime,nodiratime,discard '/dev/nvme0n1p1' '/mnt/disks/S6B0NL0T618459Z'
Dec 23 17:35:50 Tempest kernel: XFS (nvme0n1p1): Mounting V5 Filesystem
Dec 23 17:35:50 Tempest kernel: XFS (nvme0n1p1): Starting recovery (logdev: internal)
Dec 23 17:35:50 Tempest kernel: XFS (nvme0n1p1): Metadata CRC error detected at xfs_refcountbt_read_verify+0x12/0x5a [xfs], xfs_refcountbt block 0x3a382058 
Dec 23 17:35:50 Tempest kernel: XFS (nvme0n1p1): Unmount and run xfs_repair
Dec 23 17:35:50 Tempest kernel: XFS (nvme0n1p1): First 128 bytes of corrupted metadata buffer:
Dec 23 17:35:50 Tempest kernel: XFS (nvme0n1p1): metadata I/O error in "xfs_btree_read_buf_block.constprop.0+0x7a/0xc7 [xfs]" at daddr 0x3a382058 len 8 error 74
Dec 23 17:35:50 Tempest kernel: XFS (nvme0n1p1): Failed to recover leftover CoW staging extents, err -117.
Dec 23 17:35:50 Tempest kernel: XFS (nvme0n1p1): Filesystem has been shut down due to log error (0x2).
Dec 23 17:35:50 Tempest kernel: XFS (nvme0n1p1): Please unmount the filesystem and rectify the problem(s).
Dec 23 17:35:50 Tempest kernel: XFS (nvme0n1p1): Ending recovery (logdev: internal)
Dec 23 17:35:50 Tempest kernel: XFS (nvme0n1p1): Error -5 reserving per-AG metadata reserve pool.
Dec 23 17:35:50 Tempest unassigned.devices: Mount of 'nvme0n1p1' failed: 'mount: /mnt/disks/S6B0NL0T618459Z: can't read superblock on /dev/nvme0n1p1.        dmesg(1) may have more information after failed mount system call. '
Dec 23 17:38:42 Tempest unassigned.devices: Adding partition 'nvme0n1p1'...
Dec 23 17:38:42 Tempest unassigned.devices: Mounting partition 'nvme0n1p1' at mountpoint '/mnt/disks/S6B0NL0T618459Z'...
Dec 23 17:38:42 Tempest unassigned.devices: Mount drive command: /sbin/mount -t 'xfs' -o rw,noatime,nodiratime,discard '/dev/nvme0n1p1' '/mnt/disks/S6B0NL0T618459Z'
Dec 23 17:38:42 Tempest kernel: XFS (nvme0n1p1): Mounting V5 Filesystem
Dec 23 17:38:42 Tempest kernel: XFS (nvme0n1p1): Starting recovery (logdev: internal)
Dec 23 17:38:42 Tempest kernel: XFS (nvme0n1p1): Metadata CRC error detected at xfs_refcountbt_read_verify+0x12/0x5a [xfs], xfs_refcountbt block 0x3a382058 
Dec 23 17:38:42 Tempest kernel: XFS (nvme0n1p1): Unmount and run xfs_repair
Dec 23 17:38:42 Tempest kernel: XFS (nvme0n1p1): First 128 bytes of corrupted metadata buffer:
Dec 23 17:38:42 Tempest kernel: XFS (nvme0n1p1): metadata I/O error in "xfs_btree_read_buf_block.constprop.0+0x7a/0xc7 [xfs]" at daddr 0x3a382058 len 8 error 74
Dec 23 17:38:42 Tempest kernel: XFS (nvme0n1p1): Failed to recover leftover CoW staging extents, err -117.
Dec 23 17:38:42 Tempest kernel: XFS (nvme0n1p1): Filesystem has been shut down due to log error (0x2).
Dec 23 17:38:42 Tempest kernel: XFS (nvme0n1p1): Please unmount the filesystem and rectify the problem(s).
Dec 23 17:38:42 Tempest kernel: XFS (nvme0n1p1): Ending recovery (logdev: internal)
Dec 23 17:38:42 Tempest kernel: XFS (nvme0n1p1): Error -5 reserving per-AG metadata reserve pool.
Dec 23 17:38:42 Tempest unassigned.devices: Mount of 'nvme0n1p1' failed: 'mount: /mnt/disks/S6B0NL0T618459Z: can't read superblock on /dev/nvme0n1p1.        dmesg(1) may have more information after failed mount system call. '

** Press ANY KEY to close this window ** 

[trurl]

Attach diagnostics to your NEXT post in this thread

[ManBullMoose]

tempest-diagnostics-20221223-2131.zip

[trurl]

 

[ManBullMoose]

I click on the check, and it's a slightly different than what the directions show. It gives a log, and towards the bottom there are two buttons, one is "Run with correct flag" and done. Am I supposed to run it with correct flag? This is what pops up in that window.

FS: xfs

Executing file system check: /sbin/xfs_repair -n /dev/nvme0n1p1 2>&1
Phase 1 - find and verify superblock...
Phase 2 - using internal log
- zero log...
ALERT: The filesystem has valuable metadata changes in a log which is being
ignored because the -n option was used. Expect spurious inconsistencies
which may be resolved by first mounting the filesystem to replay the log.
- scan filesystem freespace and inode maps...
Metadata CRC error detected at 0x44108d, xfs_bnobt block 0x74704068/0x1000
btree block 2/1 is suspect, error -74
bad magic # 0 in btbno block 2/1
Metadata CRC error detected at 0x44108d, xfs_cntbt block 0x74704070/0x1000
btree block 2/2 is suspect, error -74
bad magic # 0 in btcnt block 2/2
Metadata CRC error detected at 0x44108d, xfs_bnobt block 0x3a382038/0x1000
btree block 1/1 is suspect, error -74
bad magic # 0 in btbno block 1/1
Metadata CRC error detected at 0x4728bd, xfs_refcountbt block 0x74704088/0x1000
btree block 2/5 is suspect, error -74
bad magic # 0 in refcount btree block 2/5
bad refcountbt block count 0, saw 1
agf_freeblks 121856122, counted 0 in ag 2
agf_longest 121856122, counted 0 in ag 2
Metadata CRC error detected at 0x44108d, xfs_cntbt block 0x3a382040/0x1000
btree block 1/2 is suspect, error -74
Metadata CRC error detected at 0x44108d, xfs_bnobt block 0xaea86098/0x1000Metadata CRC error detected at 0x46fd5d, xfs_inobt block 0x74704078/0x1000
btree block 3/1 is suspect, error -74

bad magic # 0 in btbno block 3/1
btree block 2/3 is suspect, error -74
bad magic # 0 in btcnt block 1/2
bad magic # 0 in inobt block 2/3
Metadata CRC error detected at 0x44108d, xfs_cntbt block 0xaea860a0/0x1000
btree block 3/2 is suspect, error -74
bad magic # 0 in btcnt block 3/2
Metadata CRC error detected at 0x4728bd, xfs_refcountbt block 0x3a382058/0x1000
Metadata CRC error detected at 0x46fd5d, xfs_finobt block 0x74704080/0x1000
btree block 1/5 is suspect, error -74
bad magic # 0 in refcount btree block 1/5
bad refcountbt block count 0, saw 1
btree block 2/4 is suspect, error -74
agf_freeblks 122094588, counted 0 in ag 1
bad magic # 0 in finobt block 2/4
agf_longest 122094588, counted 0 in ag 1
Metadata CRC error detected at 0x4728bd, xfs_refcountbt block 0xaea860b8/0x1000
btree block 3/5 is suspect, error -74
bad magic # 0 in refcount btree block 3/5
bad refcountbt block count 0, saw 1
agf_freeblks 122094586, counted 0 in ag 3
agf_longest 122094586, counted 0 in ag 3
Metadata CRC error detected at 0x46fd5d, xfs_inobt block 0x3a382048/0x1000
btree block 1/3 is suspect, error -74
bad magic # 0 in inobt block 1/3
Metadata CRC error detected at 0x46fd5d, xfs_finobt block 0x3a382050/0x1000
btree block 1/4 is suspect, error -74
Metadata CRC error detected at 0x46fd5d, xfs_inobt block 0xaea860a8/0x1000
btree block 3/3 is suspect, error -74
bad magic # 0 in inobt block 3/3
bad magic # 0 in finobt block 1/4
Metadata CRC error detected at 0x46fd5d, xfs_finobt block 0xaea860b0/0x1000
btree block 3/4 is suspect, error -74
bad magic # 0 in finobt block 3/4
sb_fdblocks 433178810, counted 70545081
- found root inode chunk
Phase 3 - for each AG...
- scan (but don't clear) agi unlinked lists...
- process known inodes and perform inode discovery...
- agno = 0
- agno = 1
- agno = 2
- agno = 3
- process newly discovered inodes...
Phase 4 - check for duplicate blocks...
- setting up duplicate extent list...
- check for inodes claiming duplicate blocks...
- agno = 0
- agno = 3
- agno = 1
- agno = 2
would rebuild corrupt refcount btrees.
No modify flag set, skipping phase 5
Inode allocation btrees are too corrupted, skipping phases 6 and 7
No modify flag set, skipping filesystem flush and exiting.

File system corruption detected!

[ManBullMoose]

Ok, I realized there were more pages to that thread, I will run with correct flag

[ManBullMoose]

FS: xfs

Executing file system check: /sbin/xfs_repair -e /dev/nvme0n1p1 2>&1
Phase 1 - find and verify superblock...
Phase 2 - using internal log
- zero log...
ERROR: The filesystem has valuable metadata changes in a log which needs to
be replayed. Mount the filesystem to replay the log, and unmount it before
re-running xfs_repair. If you are unable to mount the filesystem, then use
the -L option to destroy the log and attempt a repair.
Note that destroying the log may cause corruption -- please attempt a mount
of the filesystem before doing this.

Dirty log detected!
 

I now have a button that says force zero logging.

[trurl]

1 hour ago, ManBullMoose said:

force zero logging

do it

[ManBullMoose]

Awesome, that was so easy I thought it was wrong. Drive is mountable, and shows up within the vm. Thank you, Trurl. Happy Holidays to you.

[trurl]

Check to see if repair created a lost+found folder on the disk

[ManBullMoose]

Not that I can see anywhere. I've looked at the disk share through the main page by clicking on the drive itself, as well as checked my shares on the array, and I don't see any lost+found folder. Within the vm the disk shows up, and all the data I had on it is still there. I don't see anything out of the ordinary.

[itimpi]

1 hour ago, ManBullMoose said:

I don't see any lost+found folder.

This is a good sign.  Normally it means the recovery was completely successful with no data loss.