Cache drive unmountable

[dave]

HI all,

 

Yesterday my Docker containers started complaining about a storage/memory space issue. It eventually took them all down so I stopped then restarted the Docker service in an attempt to reset the state. This did not work as my Docker service was unable to start after being stopped. I then rebooted the server and now am seeing that my cache drive is unmountable saying there is no file system on it. It is btrfs formatted and has been operating fine for months. I've attached the diagnostics post-reboot. Thanks for all the help!!!

 

Dave

tower-diagnostics-20231222-0940.zip

[JorgeB]

If the log tree is the only problem this may help:

 

btrfs rescue zero-log /dev/sdf1

 

Then restart the array.

[dave]

Running a check on the device shows this:

 

root@Tower:~# btrfs check /dev/sdf1
Opening filesystem to check...
Checking filesystem on /dev/sdf1
UUID: ff7744d4-b586-4394-bf78-a469c80c4586
[1/7] checking root items
[2/7] checking extents
data extent[125475991552, 4096] referencer count mismatch (root 5 owner 32874 offset 698318848) wanted 0 have 1
data extent[125475991552, 4096] bytenr mimsmatch, extent item bytenr 125475991552 file item bytenr 0
data extent[125475991552, 4096] referencer count mismatch (root 17870283321406128133 owner 72057080812842871 offset 698318848) wanted 1 have 0
backpointer mismatch on [125475991552 4096]
ERROR: errors found in extent allocation tree or chunk allocation
[3/7] checking free space tree
[4/7] checking fs roots
[5/7] checking only csums items (without verifying data)
[6/7] checking root refs
[7/7] checking quota groups skipped (not enabled on this FS)
found 80707051520 bytes used, error(s) found
total csum bytes: 71179920
total tree bytes: 353189888
total fs tree bytes: 163463168
total extent tree bytes: 89145344
btree space waste bytes: 83147137
file data blocks allocated: 3668538724352
 referenced 64405909504

 

Edited December 22, 2023 by dave

[dave]

11 minutes ago, JorgeB said:

If the log tree is the only problem this may help:

 

btrfs rescue zero-log /dev/sdf1

 

Then restart the array.

Parity check is running right now - will try this after completion. Thanks!

[dave]

Ended up mounting the "bad" drive and copied the contents to a temporary directory. It looks like no or minimal data loss. TBD.

Currently copying over to the formatted drive pool and should be able to be up and running shortly.

 

I recall this happening in the past so I'm thinking my SSD drive(s) are flaky and need replacing. Will start that search after the new year....

 

Thanks all!