Unable to move files from cache after it completely filled up

[b01000100]

Everything has been working as expected for a long while.  Last night I was downloading a bunch of stuff, and it caused my cache to completely fill up.  I got warning emails this morning saying it was full.  I thought I could go out and delete some unnecessary things/move some files around.  When I tried to do this using Krusader, everything acts like it is in readonly mode.  

 

I generated diagnostics (attached) and looked at the syslog file.  Among everything else, I see the following two lines:

 

BTRFS: error (device sdk1) in btrfs_finish_ordered_io:2736: errno=-28 No space left

[...]

BTRFS info (device sdk1): forced readonly

 

Googling those lands me some posts saying to free up space.  I was going to copy some files to another server and delete them from this server, but the readonly won't let me delete anything after copying. 

 

Is there a way to clean everything up and get it back to normal without losing anything?

 

forwarduntodawn-diagnostics-20230101-0855.zip

[itimpi]

Not a solution to your immediate problem, but you have not set a Minimum Free Space value for any of your pools to stop them filling completely (and then bypassing the pool and writing new files to the array)

[b01000100]

Thanks for the callout.  I stopped everything that can write, and I'll be sure to set that value if I can get things fixed up (trying to leave things as-is for now until I'm sure I won't make anything worse by changing some setting somewhere).

[b01000100]

OK, so I found another forum topic that addresses this.  I have no idea how I missed it the first time searching.  Let's blame it on the celebration last night.

 

Here is the forum topic that helped me: BTRFS CACHE FULL AND READONLY

 

I disabled Docker and VMs, stopped the array, restarted the array, and then was able to run the mover.  That will clear up some space initially so that I can take care of everything else.  

 

The key component of the post linked above was this:

 

Quote

One of your NVMe devices is slightly larger than the other, this can cause issues with a full filesystem since btrfs still sees free space on one of them, but because it's raid1 it cannot write to the other one, try disabling all services that use the pool, like VMs/dockers before array start and then start the array, without any new writes the fs might not go immediately read only.

 

Sure enough, that worked.  Once again, JorgeB to the rescue.  Thank you, itimpi, for the callout of no Minimum Free Space value being set.  I'll get that set now.