long story short (as I can)...

had to take both main servers down tonight, to clean both and replace a failed cache module/battery in the other (non-unRAID box).

unRAID seemed to start up normally, and it allowed me to start the array. what I DIDN'T KNOW was it tossed one of the cache drives out of the pool, and that caused all kinds of hell.

No, I don't have a backup of the stuff stored on cache permanently (dockers, VMs, ISOs). Yes i'm a moron.

Looks like I might have found a paid tool that can help me recover some of the files, but of course the directory tree is all FUBAR.

Tentatively appears I can get the VM IMG files, as well as the docker.img and some other files.

Can someone take a tree snapshot of their 'domain' so I can look for and possibly partially reconstruct stuff to toss on the new cache i'll be obviously making?

Any specific files/folders I should try and recover to make my life easier?

I have "Community Applications appdata backup / restore module" backups ready to go with those.

Can you go into more details? First, Unraid should not autostart with a missing pool device, and even starting a pool like that should not be a big deal, if the pool wasn't redundant, it would not mount; if it was redundant, it should remove that device.

It may be worth posting the diagnostics and the output from btrfs fi show, in case the pool can still be recovered.

sadly at 0100 in the morning I wasn't thinking clearly, and didn't catch any logs.

I have my system set to NOT auto start the array on a reboot.

I manually kicked it off, and it mounted the array, minus the dockers and VMs.

that's when I started looking more in depth.

with the cache started, disk 1 in the pool was listed as unmountable, and disk 2 claimed it was unformatted.

I tried the whole array start/stop dance with removing and adding the cache drives back in to reset stuff, but that didn't work.

I tried all the btrfs repair options, even the dreaded --repair, ALL of which kicked out claiming they couldn't do anything.

so a little update.

I wound up creating a new cache pool on a pair of newer, bigger disks.

recovered my containers (mostly... shinobi still wants to give me some fits, but it's shinobi).

recreated 3 of the 4 VMs. the 4th one was my SageTV server, which was (I did manage to get it back, more in a bit) a major bummer, but we're getting around it for now, since live TV is no longer a thing here (cord thoroughly cut.)

after nearly a month and working with a company that does file recovery (their program is having fits with the btrfs under NTFS image files), I did manage to recover ALL of my VM images.

A different program that did it for me was ReclaiMe Pro build 4322.

Now... the most important thing I learned:

Since day 1, my cache pool over BTRFS was WRONG. It wasn't actually mirroring stuff correctly.

Here's what the new pool looked like until recently:

btrfs fi df /mnt/cache

Data, RAID1: total=68.00GiB, used=50.28GiB

Data, single: total=1.00GiB, used=0.00B

System, single: total=4.00MiB, used=16.00KiB

Metadata, single: total=1.01GiB, used=43.02MiB

GlobalReserve, single: total=16.00MiB, used=0.00B

see anything WRONG there?

That's the default unRAID 6.7.2 does. no idea if that's the case for newer versions.

yea... you can't recover from a single disk when btrfs has all those items in 'single' mode. I tried. Neither disk alone would allow me to recover viable data. Both could find files with the correct 'vdisk1.img' names, but none of them contained useful data when extracted.

LUCKILY for me, the corruption, while it hit both drives, didn't crash them totally. Connecting both disks to a stand-alone machine and using the aformentioned software, I was able to get it to recognize enough of the folder and data structures to successfully get it to recover the VMs, which is all I frankly needed.

Running the "balance" option from the GUI did nothing to fix the problem with the pool, even though the flags listed below are supposedly 'default'.

What *did* fix it, was in a terminal session running: btrfs balance start -dconvert=raid1 -mconvert=raid1 /mnt/cache

After some time the following can be seen:

Data, RAID1: total=51.00GiB, used=50.51GiB

System, RAID1: total=32.00MiB, used=16.00KiB

Metadata, RAID1: total=1.00GiB, used=44.41MiB

GlobalReserve, single: total=16.00MiB, used=0.00B

NOW my cache is properly protected... hopefully. I haven't yanked a disk and seen what can be found, but that sure looks better than before.

Also, and I never really understood why until now, before this I noticed that disk activity on the pool drives never seemed to be identical. NOW it is.

I know some will say I should upgrade to the latest version, and frankly I might after all this despite the problems i'll encounter, but hopefully this provides SOME kind of warning to people who are still on older versions and stuck with a btrfs formatted cache pool. Better File System my ***.

Edited November 5, 2025Nov 5 by sota