V 6.11.1, btrfs cache drive lost it's fs, during botched replacement job.

[Boojah]

Hi!

Upon starting array the cache drive says "Unmountable: Wrong or no file system", The Disk log information says:

Oct 24 10:36:11 Tore kernel: ata2: SATA max UDMA/133 abar m2048@0xf9000000 port 0xf9000180 irq 41
Oct 24 10:36:11 Tore kernel: ata2: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
Oct 24 10:36:11 Tore kernel: ata2.00: ATA-7: INTEL SSDSA2M160G2HP, 2CV102HA, max UDMA/100
Oct 24 10:36:11 Tore kernel: ata2.00: 312581808 sectors, multi 1: LBA48 NCQ (depth 32)
Oct 24 10:36:11 Tore kernel: ata2.00: configured for UDMA/100
Oct 24 10:36:11 Tore kernel: ata2.00: Enabling discard_zeroes_data
Oct 24 10:36:11 Tore kernel: sd 3:0:0:0: [sdb] 312581808 512-byte logical blocks: (160 GB/149 GiB)
Oct 24 10:36:11 Tore kernel: sd 3:0:0:0: [sdb] Write Protect is off
Oct 24 10:36:11 Tore kernel: sd 3:0:0:0: [sdb] Mode Sense: 00 3a 00 00
Oct 24 10:36:11 Tore kernel: sd 3:0:0:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
Oct 24 10:36:11 Tore kernel: sd 3:0:0:0: [sdb] Preferred minimum I/O size 512 bytes
Oct 24 10:36:11 Tore kernel: ata2.00: Enabling discard_zeroes_data
Oct 24 10:36:11 Tore kernel: sdb: sdb1
Oct 24 10:36:11 Tore kernel: sd 3:0:0:0: [sdb] Attached SCSI disk
Oct 24 10:36:11 Tore kernel: BTRFS: device fsid f313f950-f0d2-4f92-85e1-d4e4d3883935 devid 1 transid 24037583 /dev/sdb1 scanned by udevd (673)
Oct 24 10:36:53 Tore  emhttpd: INTEL_SSDSA2M160G2HP_CVPO044502UN160AGN (sdb) 512 312581808
Oct 24 10:36:53 Tore  emhttpd: import 30 cache device: (sdb) INTEL_SSDSA2M160G2HP_CVPO044502UN160AGN
Oct 24 10:36:53 Tore  emhttpd: read SMART /dev/sdb
Oct 24 10:37:19 Tore  emhttpd: INTEL_SSDSA2M160G2HP_CVPO044502UN160AGN (sdb) 512 312581808
Oct 24 10:37:20 Tore  emhttpd: read SMART /dev/sdb
Oct 24 10:42:33 Tore  emhttpd: INTEL_SSDSA2M160G2HP_CVPO044502UN160AGN (sdb) 512 312581808
Oct 24 10:42:34 Tore  emhttpd: import 30 cache device: (sdb) INTEL_SSDSA2M160G2HP_CVPO044502UN160AGN
Oct 24 10:42:34 Tore  emhttpd: read SMART /dev/sdb
Oct 24 10:57:10 Tore  emhttpd: INTEL_SSDSA2M160G2HP_CVPO044502UN160AGN (sdb) 512 312581808
Oct 24 10:57:12 Tore  emhttpd: import 30 cache device: (sdb) INTEL_SSDSA2M160G2HP_CVPO044502UN160AGN
Oct 24 10:57:12 Tore  emhttpd: read SMART /dev/sdb
Oct 24 10:57:51 Tore  emhttpd: INTEL_SSDSA2M160G2HP_CVPO044502UN160AGN (sdb) 512 312581808
Oct 24 10:57:52 Tore  emhttpd: read SMART /dev/sdb
Oct 24 10:58:05 Tore  emhttpd: shcmd (1261): /usr/sbin/cryptsetup luksErase /dev/sdb1
Oct 24 10:58:05 Tore root: Device /dev/sdb1 is not a valid LUKS device.
Oct 24 10:58:05 Tore  emhttpd: shcmd (1262): /sbin/wipefs -a /dev/sdb1
Oct 24 10:58:05 Tore root: /dev/sdb1: 8 bytes were erased at offset 0x00010040 (btrfs): 5f 42 48 52 66 53 5f 4d
Oct 24 11:01:42 Tore  emhttpd: read SMART /dev/sdb
Oct 24 11:02:05 Tore  emhttpd: INTEL_SSDSA2M160G2HP_CVPO044502UN160AGN (sdb) 512 312581808
Oct 24 11:02:05 Tore  emhttpd: import 30 cache device: (sdb) INTEL_SSDSA2M160G2HP_CVPO044502UN160AGN
Oct 24 11:02:05 Tore  emhttpd: read SMART /dev/sdb
Oct 24 11:03:52 Tore  emhttpd: shcmd (1516): mount -t btrfs -o noatime,space_cache=v2 /dev/sdb1 /mnt/cache
Oct 24 11:03:52 Tore root: mount: /mnt/cache: wrong fs type, bad option, bad superblock on /dev/sdb1, missing codepage or helper program, or other error.
Oct 24 11:07:56 Tore  emhttpd: read SMART /dev/sdb
Oct 24 11:09:00 Tore  emhttpd: INTEL_SSDSA2M160G2HP_CVPO044502UN160AGN (sdb) 512 312581808

 

What happened beforehand:

I have one old ssd cache drive (on btrfs) that I tried to replace. I tried to follow: 

 

What I did:

1. Clean shutdown.

2. Installed the new nvme ssd on pcie adapter card.

3. Boot, now in unraid it did not start the array, too many drives for my license. So I upgraded.

4. The old ssd cache drive was replaced with the new one in the dropdown list.

5. Started the array, nvme drive needed fs, so i formatted.

6. Started up fine, but now I realize that it will not magically copy contents from the old drive as it is not in the pool any longer...

7. Shutdown array.

8. Put the old drive back into the cache drive slot, and started.

9. Logs as above, wont start the cache pool.

 

So I didn't read the crossed over part properly in the post above, and assumed this metod was valid for 1 drive pools... Yeah I know... Also Wiki page on this topic that was linked when I searched the forum was not available at the time or it was old links.

 

Anyway I would imagine the data is still there, so would it be possible to fix the drive and do a mirror/copy to the new drive?

 

I do have a "CA Backup / Restore Appdata" backup that was done the day before, the mover had been run too, so really should be no files missing. So I suppose the most straightforward way would be to restore the backup and reconfigure the dockers.

What would be fastest/easiest? It would be fun to fix the old drive, if just to understand what went wrong and learn something from this.

I hope the diagnostics zip contains all re rest of the required informations.

 

Thanks!

Boojah-diagnostics-20221025-2202.zip

[JorgeB]

Stope the array, type:

 

btrfs-select-super -s 1 /dev/sdb1

 

Start array, post new diags.

[Boojah]

Ok I did that and now it works like before, thanks!

Here are the diags.

Looks ok or should I take it down and prepare a swap?

What method is recommended for single drive swaps? Mover method? https://wiki.unraid.net/Manual/Storage_Management#Backing_up_the_pool_to_the_array

That wiki does not really specify what to do on a single device pool.

 

Thanks a lot!

Boojah new-diagnostics-20221028-1919.zip

[JorgeB]

29 minutes ago, Boojah said:

What method is recommended for single drive swaps? Mover method? https://wiki.unraid.net/Manual/Storage_Management#Backing_up_the_pool_to_the_array

Yes, this will work.