Array failure - all drives unmountable and require xfs_repair 6.9.0-RC2

[Rohime]

Hi All,

 

Looking for some help please, before i die of panic 

 

Here's my story:

Long term unraid user with - New server build - relocated installed unraid and drives to new chassis about 2 weeks ago.

Needed to upgrade to 6.9.0-RC2 at that time because of the NIC on the motherboard - needed 6.9 for NIC driver.

 

Two weeks later - decided to replace external USB drive with internal disk (had more slots in new case).

Copied all the data off the external drive, and the was going through the process to replace the 4TB external with a 4TB internal.

All looking good - swapped in the new drive, restarted the array, it was doing parity rebuild and I went to bed.

The next morning the server was powered off.   Not expected - feeling of dread ... 

 

Rebooted the server.   Everything LOOKED ok (without any serious checking).

Restarted the Array.

All the internal drives failed to mount.    In looking at the diagnostics they all have some form of mount failure as per the below (they are not all identical).

The array did a parity rebuild with the only remaining external drive (single remaining volume).

 

Further checking then shows S.M.A.R.T. errors on all 8 x Internal SATA drives.

These devices are all connected via the same SAS/SATA HBA -- a recently purchased second hand card for this build

IBM SAS9201-8i PCIe x8 6Gbps Dual SAS/SATA 9201-8I RAID Card

 

The SMART error of concern is high     UDMA CRC error count    .

as above - its consistent across all 8 drives, two ports, two cables, two SATA backplanes - 1 card.

 

So yes - i've ordered a replacement card.   Waiting for its arrival.

 

So my questions / request for assistance are:

1.  Should i bother to do anything before i replace the SATA HBA -- (which seems to be working normally right now, if i ignore the SMART errors).

2. Are there other considerations ... or helpful hints for BEFORE i run the xfs_repair?  (which might be a while depending on timing of new card?)

3. Is this actually recoverable - or am i royally screwed?

 

Anything else i need to know to get recovered?

 

-----------------------

 

 

 

 

**Sample log entry -- variations on this for all internal drives.

 

Feb 18 15:21:25 NAS2 emhttpd: shcmd (2564): mkdir -p /mnt/disk2
Feb 18 15:21:25 NAS2 emhttpd: shcmd (2565): mount -t xfs -o noatime /dev/md2 /mnt/disk2
Feb 18 15:21:25 NAS2 kernel: XFS (md2): Mounting V5 Filesystem
Feb 18 15:21:25 NAS2 kernel: XFS (md2): Corruption warning: Metadata has LSN (1:1031651) ahead of current LSN (1:1031638). Please unmount and run xfs_repair (>= v4.3) to resolve.
Feb 18 15:21:25 NAS2 kernel: XFS (md2): log mount/recovery failed: error -22
Feb 18 15:21:25 NAS2 kernel: XFS (md2): log mount failed
Feb 18 15:21:25 NAS2 root: mount: /mnt/disk2: wrong fs type, bad option, bad superblock on /dev/md2, missing codepage or helper program, or other error.
Feb 18 15:21:25 NAS2 emhttpd: shcmd (2565): exit status: 32
Feb 18 15:21:25 NAS2 emhttpd: /mnt/disk2 mount error: not mounted
Feb 18 15:21:25 NAS2 emhttpd: shcmd (2566): umount /mnt/disk2
Feb 18 15:21:25 NAS2 root: umount: /mnt/disk2: not mounted.
Feb 18 15:21:25 NAS2 emhttpd: shcmd (2566): exit status: 32
Feb 18 15:21:25 NAS2 emhttpd: shcmd (2567): rmdir /mnt/disk2
 

nas2-diagnostics-20210218-1527.zip

[JorgeB]

Run a filesystem check (without -n) on those drives, if the emulated disk5 is not fixable or is severely corrupt it could be better to re-sync parity with the old disk then upgrade again.