(SOLVED) Problem with cache pool

[Esg450]

Hi, while I was updating a few docker containers, my entire docker process froze up and I wasn't able to start or stop any containers. I attempted to delete and recreate the docker image folder, only to get "Read-only system" errors.

 

I then attempted the solution provided here with no luck (ERROR: error during balancing '/mnt/cache': Read-only file system): 

 

I checked my cache disk log and saw the following errors as well, but haven't been able to find any solutions to attempt to troubleshoot these:

Jun 18 17:25:40 MediaServer kernel: BTRFS error (device sdg1): unable to find ref byte nr 4513148928 parent 0 root 1456 owner 4872 offset 0
Jun 18 17:25:40 MediaServer kernel: BTRFS: error (device sdg1) in __btrfs_free_extent:3092: errno=-2 No such entry
Jun 18 17:25:40 MediaServer kernel: BTRFS info (device sdg1): forced readonly
Jun 18 17:25:40 MediaServer kernel: BTRFS: error (device sdg1) in btrfs_run_delayed_refs:2144: errno=-2 No such entry

 

Here is the result of my brtfs check as well:

[1/7] checking root items
[2/7] checking extents
data backref 2706669568 root 1485 owner 10272 offset 0 num_refs 0 not found in extent tree
incorrect local backref count on 2706669568 root 1485 owner 10272 offset 0 found 1 wanted 0 back 0xdcc8450
incorrect local backref count on 2706669568 root 18446612687375494800 owner 10272 offset 0 found 0 wanted 1 back 0x4b78cd0
backref disk bytenr does not match extent record, bytenr=2706669568, ref bytenr=0
backpointer mismatch on [2706669568 4096]
data backref 4513148928 root 1456 owner 4872 offset 0 num_refs 0 not found in extent tree
incorrect local backref count on 4513148928 root 1456 owner 4872 offset 0 found 1 wanted 0 back 0xc931ca0
incorrect local backref count on 4513148928 root 10952754293765047728 owner 72057080910090472 offset 0 found 0 wanted 1 back 0xc814310
backref disk bytenr does not match extent record, bytenr=4513148928, ref bytenr=0
backpointer mismatch on [4513148928 12288]
ERROR: errors found in extent allocation tree or chunk allocation
[3/7] checking free space tree
[4/7] checking fs roots
root 1499 inode 7288 errors 200, dir isize wrong
root 1499 inode 14372 errors 1, no inode item
	unresolved ref dir 7288 index 16 namelen 24 name update-notifier-npm.json filetype 1 errors 5, no dir item, no inode ref
ERROR: errors found in fs roots
Opening filesystem to check...
Checking filesystem on /dev/sdg1
UUID: d5eaeb34-7231-4d19-b745-ff022a4bb3ac
cache and super generation don't match, space cache will be invalidated
found 148873519104 bytes used, error(s) found
total csum bytes: 37440396
total tree bytes: 1119322112
total fs tree bytes: 1001799680
total extent tree bytes: 59899904
btree space waste bytes: 229142372
file data blocks allocated: 229980651520
 referenced 151732486144

 

I've attached my server's diagnostics file, but let me know if there's anything else I can try.

 

Thanks in advanced!

mediaserver-diagnostics-20210618-1730.zip

Edited June 18, 2021 by Esg450

[Esg450]

I just solved this issue using btrfs repair. This was my last resort before going for a full reformat of the cache pool, so I would recommend attempting everything else first and also backing up your cache pool before attempting this.

 

I started the array in maintenance mode, then ran the following command (targeting my specific device, /dev/sdg1):

btrfs check --repair /dev/sdg1

 

It seems like it worked and everything is back to normal including my docker containers. Again it seems like this isn't a tested method, so use with caution and ensure you have a backup before attempting.