Looks as the filesystem is corrupted on my NVMe cache pool drive. Formatted as btfs, and contains my system share (docker and libvrt images). Was finally able to get the array to stop (wouldn't unmount the cache drive, so needed to "umount -l /dev/loop2").

Restarted the array in Maintenance mode, and ran Check Filesystem Status (--readonly). Results below, as well as a diagnostic run after that. Short SMART test also run, no issues there).

What is my next step in getting this server back up and running? TAI

[1/8] checking log skipped (none written)
[2/8] checking root items
[3/8] checking extents
ref mismatch on [183065169920 8192] extent item 1, found 0
data extent[183065169920, 8192] bytenr mimsmatch, extent item bytenr 183065169920 file item bytenr 0
data extent[183065169920, 8192] referencer count mismatch (root 5 owner 20159943 offset 2510848) wanted 1 have 0
backpointer mismatch on [183065169920 8192]
owner ref check failed [183065169920 8192]
ERROR: invalid generation for extent 183065362432, have 288230376153960935 expect (0, 2249226]
ref mismatch on [183065362432 147456] extent item 1, found 2
data extent[183065362432, 147456] bytenr mimsmatch, extent item bytenr 183065362432 file item bytenr 183065432064
data extent[183065362432, 147456] size mimsmatch, extent item size 147456 file item size 8192
data extent[183065362432, 147456] referencer count mismatch (root 5 owner 20159943 offset 2510848) wanted 0 have 1
backpointer mismatch on [183065362432 147456]
ref mismatch on [1194265329664 16384] extent item 144115188075855873, found 1
tree extent[1194265346048, 16384] root 5 has no backref item in extent tree
tree extent[1194265346048, 16384] root 144115188075855877 has no tree block found
incorrect global backref count on 1194265346048 found 2 wanted 1
backpointer mismatch on [1194265346048 16384]
ERROR: errors found in extent allocation tree or chunk allocation
[4/8] checking free space tree
We have a space info key for a block group that doesn't exist
[5/8] checking fs roots
ERROR: unexpected condition, probably corruption: node refs 1 <= 1
check/main.c:2063: walk_up_tree: BUG_ON `ret` triggered, value -117
/sbin/btrfs() [0x492fe9]
/sbin/btrfs() [0x49dc73]
/sbin/btrfs(main+0x92) [0x411ee2]
/lib64/libc.so.6(+0x2ad14) [0x15060f8c0d14]
/lib64/libc.so.6(__libc_start_main+0x86) [0x15060f8c0dc6]
/sbin/btrfs(_start+0x21) [0x4136e1]

malta-tower-diagnostics-20251122-0931.zip

Adding extended SMART - No errors reported.

malta-tower-smart-20251122-1015.zip

One more Diagnostics, with array started in normal (not Maintenance mode) and Docker/VM set to not auto-start.

malta-tower-diagnostics-20251122-1047.zip

Tried running Scrub, but it aborts immediately after starting (tried both in GUI and CLI)

Drive seems to be running read only - cannot right to it (both using Uraid or SMB)

Any advice before I blow this away and attempting to reformat it?

I ended up reformatting the drive. Tried a btrfs check --repair. The drive was then writable, but I ran a scrub afterward and it found several unrecoverable errors. Extended SMART still shows no errors, so I just restored/rebuilt everything. I wanted to do a cleanup anyway for there are a number of dockers and VMs I no longer use, so I guess there was something of value that came out of this.

8 hours ago, ConnerVT said:

so I just restored/rebuilt everything

That's the best bet considering the filesystem damage, it may also be a good idea to run memtest or the RAM plugin tester, to see if there are any obvious RAM issues.

@JorgeB Thanks for the feedback.

Running a memtest is not a bad idea. Just difficult to find the down time to do so, as several family and friends use my media server, and I get "Why can't I watch my show..." whining when the server is offline. ☹️

I believe the corruption may have happened during a new VM I have been fighting to install. I'll accept this event as a one off occurrence, and keep an eye on things for any strange behavior.

It was actually a good experience, for it allowed me to test my backup strategy. I knew that burning and rebuilding was always an option, but was interested to see if it could have been repaired. The NVMe holds my system (docker.img, libvrt.img and some associated scripts) and appdata share, and a Downloads share that only holds files before quickly moved to the array once completed. So there really wasn't any danger in losing valuable data. As long as my backup plan actually restores properly. I now know it will.

It took about an hour for me to have the docker applications all up and running (20 running, 5 stopped utilities). I was pleasantly surprised that I didn't need to do any post-restoration configuration changes.

1 hour ago, ConnerVT said:

Just difficult to find the down time to do so

You can use the live memory tester for Unraid plugin, it may not be able to test as much RAM as memtest, but I've seen it find errors multiple times.

This has already been answered but just in case someone comes here from a search engine looking for info on the We have a space info key for a block group that doesn't exist message (unrelated to OP's issue) - don't worry. I think something has changed recently in btrfs because I got it on a fresh Ubuntu install on an NVMe and have noticed it multiple times on multiple drives, HDD or not, when trying to resize them with gparted. You can run a scrub if you are worried.

Running either btrfs-rescue clear-space-cache (v1|v2) /dev/xxx or btrfs-check --clear-space-cache (v1|v2) /dev/xxx. It seems Sorry for the somewhat unrelated reply, this was one of the first threads that came up when I was looking into the issue.