NVMe drive disappeared, cache pool now broken

[LumpyCustard]

I upgraded my server today -- Replaced the motherboard, CPU, and added additional RAM. When i came home i shut down UNRAID gracefully, however after a full graceful shutdown my server LED was still on and the fans were running. I checked the monitor connected to my server and it was completely blank and unresponsive. After waiting 15 minutes i force shut down the server by holding the power button.

 

I upgraded the server and put everything back together, booted into BIOS, updated to the latest firmware as of today (Gigabyte Z690 Gaming X - BIOS version F22), enabled virtualisation on the CPU, turned on XMP, and booted into UNRAID.

 

Unfortunately 1 of my 4 cache drives was not detected. I reseated everything, rebooted multiple times, but it's simply not detected, even in BIOS. I'm wondering if the forced shutdown corrupted the drive's firmware.

 

I inserted the drive into an NVMe caddy and connected it to my Windows PC and it's unable to read the drive, stating that it is uninitialized. I haven't attempted to initialize it because i don't want to nuke the data that's on it. I connected a known working NVMe from UNRAID to the same caddy and it was immediately detected and showed a healthy 1TB partition in Windows.

 

I've tried cleaning the gold contacts on the NVMe with electrical PCB cleaner as well, but that did not work.

 

My cache was set up as a 4 drive BTRFS pool (balanced? striped? not sure of the terminology)

 

I attempted following advice from some of the moderators on this forum after hours of searching, including starting the array with no cache drives to forget the config then starting it again with the drives, but unfortunately my cache drives/shares are not mountable. I got a bunch of errors from UNRAID about missing shares and files, and my Docker containers and VM's are all missing.

 

My "pool devices" was set to 4 devices, and UNRAID was complaining that a drive was missing / config not valid. I made the mistake of clicking "3 slots", then it immediately dropped the drive from the configuration. When i selected "4" slots again, UNRAID was no longer complaining about invalid configuration.

 

 

 

I'm unsure how a BTRFS pool works, but if it is striped then i assume my data is not recoverable.

 

I am willing to pay for data recovery from a local reputable data recovery company that specialises in flash storage, however i want to know:

 

A) If anything is currently salvageable. My hope is that the 250GB virtual machine lived on one SSD and that i'd be able to recover it by mounting that SSD and copying the virtual disk file. My docker containers are backed up, all i care about is my virtual machine.

 

B) If i do take the NVMe to a data recovery company and they manage to fix it, have I completely nuked my cache drive by starting it with 3 drives instead of 4? I DID NOT choose the option to format when starting the array.

 

Diagnostics attached.

 

Thank you in advance.

devoraid-diagnostics-20230205-2026.zip

Edited February 5, 2023 by LumpyCustard

[JorgeB]

Post the output of:

btrfs fi show

 

[LumpyCustard]

Thank you so much for the reply. Here is the output (array not running)

 

Edited February 5, 2023 by LumpyCustard

[JorgeB]

OK, now start the array with all 3 devices assigned to a new pool, i.e., there can't be a "data on this device will be deleted" warning in front of any of the pool devices, then post new diags.

[LumpyCustard]

OK just to make sure i understand since i'm stressing about the data, you want me to take the array offline, create an entirely new pool, then assign the 3 alive SSD's to that new pool then start the array in maintenance mode?

 

This is what i'm looking at:

 

Edited February 5, 2023 by LumpyCustard

[JorgeB]

3 minutes ago, LumpyCustard said:

create an entirely new pool, then assign the 3 alive SSD's to that new pool then start the array in maintenance mode?

No need for a new pool, just not a pool where it's detecting missing/replaced devices, you can start the array with all pool members unassigned, Unraid will forget the pool config, them stop the array, re-assign all 3 pool members to the same pool and start the array in normal mode.

[LumpyCustard]

I ended up creating a new pool just to make sure i wasn't doubling up. See attached. This is what i'm looking at now.

 

Thank you for your help thus far.

 

devoraid-diagnostics-20230205-2312.zip

[JorgeB]

Feb  5 23:10:31 devoraid kernel: BTRFS warning (device sdc1): chunk 4163153952768 missing 1 devices, max tolerance is 0 for writable mount

This means your pool was not redundant, so it cannot mount read/write with a missing device, you can manually mount it degraded and read only but most files will likely be damaged or incomplete, if you want to try see how to mount manually below:

 

https://forums.unraid.net/topic/46802-faq-for-unraid-v6/?do=findComment&comment=543490

 

You'd need to use degraded,ro as mount options.

[LumpyCustard]

Thank you, was attempting to follow that but i wasn't sure what to point toward as the mount point. sdc1?

 

edit: ok seems like there is some data in there. Just need to enable SFTP and copy it out i suppose?

 

Edited February 5, 2023 by LumpyCustard

[JorgeB]

You can use any of the pool members, partition must be specified, e.g., /dev/sdc1

[LumpyCustard]

Looks like i'm screwed and my only lifeline will be fixing the dead NVME drive via data recovery if that even works to begin with. Serves me right for going with 0 redundancy cache pool.

 

Appreciate all your assistance.

 

 

 

Edited February 5, 2023 by LumpyCustard

[LumpyCustard]

Just thought i'd update this post. I got my drive professionally repaired by a data recovery company -- they corrected an issue on the PCB of the m.2 SSD drive.

 

Despite starting and stopping the array dozens of times, creating new cache pools, swapping the drive order around in UNRAID and so on, when i inserted the drive back in and selected the drives, my cache pool came back to life and all the data was intact -- docker started, my VM's launched, etc.

 

I am in the process of breaking down my cache pool now -- i've purchased 4 new NVMe's and i'll be rebuilding it with redundancy as well as ditching the 2.5" drives i had mixed in with NVMe drives.

 

Edited February 7, 2023 by LumpyCustard