Hi everyone. I have messed up pretty bad and was hoping to get some direction. I've been using unraid for a couple of years and I'm comfortable but by no means an expert.

 

Steps so far:

• Originally had a single SSD as a cache drive
• Installed a new SSD (different model, same size)
• Realised that the drives had been pooled rather than increasing the storage by 1TB as I intended
• Somehow changed the settings to set the drives as 'Single Device' rather than a multi pool device (I thought this was fine and would increase my capacity, but now I guess it really isn't fine)
• Carried on my day, set up some new qcow2 vdisks for my gaming VMs. Had issues of games crashing, thought it was GPU related but finally realised that it was probably drive related as it also happened when downloading games.
• Removed second cache drive from the 'pool' (although I guess technically not a pool as it was set to 'Single Device' mode?.
• Lost access to my cache completely with incredibly important data on (my daily files are stored on the cache but backed up to the array too). Unfortunately, upon checking my backups I see that for whatever reason duplicacy hasn't run for over a month so I really, really need to get back access. I have my dockers/domains backed up on the array and they should be fine.
• Can't get into any Krusader or any docker apps to try and see what data is actually left on the cache drives.
• Tried adding the second drive back, but no luck, says it needs formatting (I didn't)
• It was at this point I stopped before causing any more damage... I don't think anything was actually written to the second drive as the capacity of the cache pool never went over the size of the first disk (around 800GB out of 1TB at the max)

 

Current situation:

• Original cache drive is still in the cache pool - but it says Unmountable: no file system'
• Second cache drive is in unassigned drives
• I'm realising now that I should have just added the second SSD as an unassigned device ( I actually did this first but then changed my mind.. )

 

Please does anyone have any advice of where to go from now? I can only assume that somehow setting the pool to 'Single Device' has severely messed things up and I really have no clue how to reverse the damage. I likely don't need to recover any data from the second ssd as it was only fitted today and the only thing I've really done today is install games. But I do need to get back full access to the original cache drive.

 

 

andromeda-diagnostics-20210626-2335.zip

Edited June 26, 20215 yr by andeekaii

Error log from original cache drive:

Jun 26 22:29:09 Andromeda emhttpd: Samsung_SSD_970_EVO_Plus_1TB_S4EWNMFN804954K (nvme0n1) 512 1953525168
Jun 26 22:29:09 Andromeda emhttpd: import 30 cache device: (nvme0n1) Samsung_SSD_970_EVO_Plus_1TB_S4EWNMFN804954K
Jun 26 22:29:29 Andromeda emhttpd: Samsung_SSD_970_EVO_Plus_1TB_S4EWNMFN804954K (nvme0n1) 512 1953525168
Jun 26 22:29:29 Andromeda emhttpd: import 30 cache device: (nvme0n1) Samsung_SSD_970_EVO_Plus_1TB_S4EWNMFN804954K
Jun 26 22:30:20 Andromeda emhttpd: Samsung_SSD_970_EVO_Plus_1TB_S4EWNMFN804954K (nvme0n1) 512 1953525168
Jun 26 22:30:20 Andromeda emhttpd: import 30 cache device: (nvme0n1) Samsung_SSD_970_EVO_Plus_1TB_S4EWNMFN804954K
Jun 26 22:30:48 Andromeda kernel: BTRFS info (device nvme0n1p1): turning on async discard
Jun 26 22:30:48 Andromeda kernel: BTRFS info (device nvme0n1p1): allowing degraded mounts
Jun 26 22:30:48 Andromeda kernel: BTRFS info (device nvme0n1p1): using free space tree
Jun 26 22:30:48 Andromeda kernel: BTRFS info (device nvme0n1p1): has skinny extents
Jun 26 22:30:48 Andromeda kernel: BTRFS warning (device nvme0n1p1): devid 2 uuid b9ae0a19-3f76-4397-97b2-15a989e9472c is missing
Jun 26 22:30:48 Andromeda kernel: BTRFS warning (device nvme0n1p1): chunk 2949533270016 missing 1 devices, max tolerance is 0 for writable mount
Jun 26 22:30:48 Andromeda kernel: BTRFS warning (device nvme0n1p1): writable mount is not allowed due to too many missing devices
Jun 26 22:30:48 Andromeda root: mount: /mnt/cache: wrong fs type, bad option, bad superblock on /dev/nvme0n1p1, missing codepage or helper program, or other error.
Jun 26 22:30:48 Andromeda kernel: BTRFS error (device nvme0n1p1): open_ctree failed
Jun 26 22:31:48 Andromeda emhttpd: Samsung_SSD_970_EVO_Plus_1TB_S4EWNMFN804954K (nvme0n1) 512 1953525168
Jun 26 22:31:49 Andromeda emhttpd: import 30 cache device: (nvme0n1) Samsung_SSD_970_EVO_Plus_1TB_S4EWNMFN804954K
Jun 26 22:32:32 Andromeda emhttpd: Samsung_SSD_970_EVO_Plus_1TB_S4EWNMFN804954K (nvme0n1) 512 1953525168
Jun 26 22:32:32 Andromeda emhttpd: import 30 cache device: (nvme0n1) Samsung_SSD_970_EVO_Plus_1TB_S4EWNMFN804954K
Jun 26 22:32:50 Andromeda emhttpd: Samsung_SSD_970_EVO_Plus_1TB_S4EWNMFN804954K (nvme0n1) 512 1953525168
Jun 26 22:32:50 Andromeda emhttpd: import 30 cache device: (nvme0n1) Samsung_SSD_970_EVO_Plus_1TB_S4EWNMFN804954K
Jun 26 22:33:38 Andromeda emhttpd: Samsung_SSD_970_EVO_Plus_1TB_S4EWNMFN804954K (nvme0n1) 512 1953525168
Jun 26 22:33:38 Andromeda emhttpd: import 30 cache device: (nvme0n1) Samsung_SSD_970_EVO_Plus_1TB_S4EWNMFN804954K
Jun 26 22:33:48 Andromeda kernel: BTRFS info (device nvme0n1p1): turning on async discard
Jun 26 22:33:48 Andromeda kernel: BTRFS info (device nvme0n1p1): allowing degraded mounts
Jun 26 22:33:48 Andromeda kernel: BTRFS info (device nvme0n1p1): using free space tree
Jun 26 22:33:48 Andromeda kernel: BTRFS info (device nvme0n1p1): has skinny extents
Jun 26 22:33:48 Andromeda kernel: BTRFS warning (device nvme0n1p1): devid 2 uuid b9ae0a19-3f76-4397-97b2-15a989e9472c is missing
Jun 26 22:33:48 Andromeda kernel: BTRFS warning (device nvme0n1p1): chunk 2949533270016 missing 1 devices, max tolerance is 0 for writable mount
Jun 26 22:33:48 Andromeda kernel: BTRFS warning (device nvme0n1p1): writable mount is not allowed due to too many missing devices
Jun 26 22:33:48 Andromeda root: mount: /mnt/cache: wrong fs type, bad option, bad superblock on /dev/nvme0n1p1, missing codepage or helper program, or other error.
Jun 26 22:33:48 Andromeda kernel: BTRFS error (device nvme0n1p1): open_ctree failed
Jun 26 23:11:15 Andromeda emhttpd: Samsung_SSD_970_EVO_Plus_1TB_S4EWNMFN804954K (nvme0n1) 512 1953525168
Jun 26 23:11:15 Andromeda emhttpd: import 30 cache device: (nvme0n1) Samsung_SSD_970_EVO_Plus_1TB_S4EWNMFN804954K
Jun 26 23:23:46 Andromeda kernel: BTRFS info (device nvme0n1p1): turning on async discard
Jun 26 23:23:46 Andromeda kernel: BTRFS info (device nvme0n1p1): allowing degraded mounts
Jun 26 23:23:46 Andromeda kernel: BTRFS info (device nvme0n1p1): using free space tree
Jun 26 23:23:46 Andromeda kernel: BTRFS info (device nvme0n1p1): has skinny extents
Jun 26 23:23:46 Andromeda kernel: BTRFS warning (device nvme0n1p1): devid 2 uuid b9ae0a19-3f76-4397-97b2-15a989e9472c is missing
Jun 26 23:23:46 Andromeda kernel: BTRFS warning (device nvme0n1p1): chunk 2949533270016 missing 1 devices, max tolerance is 0 for writable mount
Jun 26 23:23:46 Andromeda kernel: BTRFS warning (device nvme0n1p1): writable mount is not allowed due to too many missing devices
Jun 26 23:23:46 Andromeda root: mount: /mnt/cache: wrong fs type, bad option, bad superblock on /dev/nvme0n1p1, missing codepage or helper program, or other error.
Jun 26 23:23:46 Andromeda kernel: BTRFS error (device nvme0n1p1): open_ctree failed

 

Error log from second cache drive:

un 26 22:29:09 Andromeda emhttpd: CT1000P2SSD8_2115E59575DF (nvme1n1) 512 1953525168
Jun 26 22:29:09 Andromeda emhttpd: import 31 cache device: (nvme1n1) CT1000P2SSD8_2115E59575DF
Jun 26 22:29:29 Andromeda emhttpd: CT1000P2SSD8_2115E59575DF (nvme1n1) 512 1953525168
Jun 26 22:29:29 Andromeda emhttpd: import 31 cache device: (nvme1n1) CT1000P2SSD8_2115E59575DF
Jun 26 22:30:20 Andromeda emhttpd: CT1000P2SSD8_2115E59575DF (nvme1n1) 512 1953525168
Jun 26 22:30:45 Andromeda emhttpd: shcmd (436): /usr/sbin/cryptsetup luksErase /dev/nvme1n1p1
Jun 26 22:30:46 Andromeda root: Device /dev/nvme1n1p1 is not a valid LUKS device.
Jun 26 22:30:46 Andromeda emhttpd: shcmd (437): /sbin/wipefs -a /dev/nvme1n1p1
Jun 26 22:30:46 Andromeda root: /dev/nvme1n1p1: 8 bytes were erased at offset 0x00010040 (btrfs): 5f 42 48 52 66 53 5f 4d
Jun 26 22:30:50 Andromeda unassigned.devices: Don't spin down device '/dev/nvme1n1'.
Jun 26 22:31:48 Andromeda emhttpd: CT1000P2SSD8_2115E59575DF (nvme1n1) 512 1953525168
Jun 26 22:31:49 Andromeda emhttpd: import 31 cache device: (nvme1n1) CT1000P2SSD8_2115E59575DF
Jun 26 22:32:32 Andromeda emhttpd: CT1000P2SSD8_2115E59575DF (nvme1n1) 512 1953525168
Jun 26 22:32:50 Andromeda emhttpd: CT1000P2SSD8_2115E59575DF (nvme1n1) 512 1953525168
Jun 26 22:32:50 Andromeda emhttpd: import 31 cache device: (nvme1n1) CT1000P2SSD8_2115E59575DF
Jun 26 22:33:38 Andromeda emhttpd: CT1000P2SSD8_2115E59575DF (nvme1n1) 512 1953525168
Jun 26 22:33:50 Andromeda unassigned.devices: Don't spin down device '/dev/nvme1n1'.
Jun 26 23:11:15 Andromeda emhttpd: CT1000P2SSD8_2115E59575DF (nvme1n1) 512 1953525168
Jun 26 23:11:15 Andromeda emhttpd: import 31 cache device: (nvme1n1) CT1000P2SSD8_2115E59575DF

 

BTRFS check of original drive:

Update: I was running the commands on the drive, rather than the partition. Running the commands on the partition is successful so I am currently restoring the data to my array and then I will attempt to repair it.

 

After a restless night I was back at it again this morning.

In my panic to get it working this morning I re-added the second drive to my pool and started it (but as I want in full screen I think I missed the notice that said the drive would be formatted). So I'm not sure if it did or didn't format the second drive...

 

But anyway, it looks like it's beyond help, none of the BTRFS recovery options here work.

Quote

wrong fs type, bad option, bad superblock on /dev/nvme0n1, missing codepage or helper program, or other error.

 

Does anyone have any idea at all before I accept the fact that I've lost 6 weeks of incredibly important business data? 

My next step will be to try some data recovery tools to at least try to pull some of my more recent files off. 

 

I'm absolutely gutted, but a reminder to you all to make backups and ideally don't store important data on the cache pool

 

Thanks

Edited June 27, 20215 yr by andeekaii

10 hours ago, andeekaii said:

• Removed second cache drive from the 'pool' (although I guess technically not a pool as it was set to 'Single Device' mode?.
• Lost access to my cache completely

You can't remove drives from a pool using single profile with the GUI, only manually.

 

You can try this on the removed drive (ckeck that it's still the correct device):

 

btrfs-select-super -s 1 /dev/nvme1n1p1

 

Then reset the pool config in Unraid:

 

Unassign all cache devices, start array to make Unraid "forget" current cache config, stop array, reassign both cache devices (there can't be an "All existing data on this device will be OVERWRITTEN when array is Started" warning for any cache device), start array and see if it mounts.

1 hour ago, JorgeB said:

You can't remove drives from a pool using single profile with the GUI, only manually.

 

Thank you Jorge. I wasn't aware of that but I will certainly remember that in the future.

 

Luckily I was already able to retrieve all of my data using the 'btrfs restore' command suggested in your FAQ post, so I am very thankful for that.

Something funky was happening with the new cache drive anyway so it's probably a good thing to format the drives and start again anyway.