Docker Service unable to start and Cache Drive Read Only

[TheSquigglyline]

I logged onto my server after one of my nextcloud users reported upload errors to find that my docker service was stopped. Upon trying to start it I was met with a "Docker service unable to start" message. After running fix common problems it found that my cache drive was read only. I made sure the docker service was off, restarted the whole server, came back the next day due to not enough time in the day, when I checked back in the cache drive was writable again and the error was gone in Fix common problems. before starting the docker service I searched the forums a bit more and came across this (https://forums.unraid.net/topic/69187-cache-marked-as-read-only-or-full-solved/), post and attempted to run the BTRFS command 

btrfs balance start -dusage=75 /mnt/cache

mentioned in this post(https://forums.unraid.net/topic/62230-out-of-space-errors-on-cache-drive/?tab=comments#comment-610551) linked from the previous.
I did have to start smaller on the dusage=. first at 50 then up to 75 slowly. 
i also ran 

btrfs filesystem usage /mnt/cache

 and my cache didn't show full. 
So I started up docker again. And it all fell apart. Dockers wouldn't starts with error that cache drive is read only. Via winSCP i tried to create a directory on the cache and got 

General failure (server should provide error description).
Error code: 4
Error message from server: Failure

Common reasons for the Error code 4 are:
- Renaming a file to a name of already existing file.
- Creating a directory that already exists.
- Moving a remote file to a different filesystem (HDD).
- Uploading a file to a full filesystem (HDD).
- Exceeding a user disk quota.

 

I really don't want to have to format my cache drive. 
I also don't know enough to know what deleting my docker.img file would do to my existing dockers. If i backed up my appdata folder will i be good? Will i have to re download and re configure all my dockers?
I have about 118gb used on my 500gb ssd cache so moving it to the array and back wouldn't be terrible but wouldn't be the most fun.

 

How did this happen. I did receive and error that my docker drive was near full. Is that the cause. I did expand it but only after these issues arose. Is it an issue of my cache drive getting too full. Sometimes when I am DLing new media for the Plex server the cache will get quite full before moving it that night. 

Attached is a copy of my diagnostics. I am unfortunetly not technical enough to solve this issue on my own. Any help would be amazing. Thank you in advance!

tower-diagnostics-20210617-1209.zip

[JorgeB]

Cache filesystem is corrupt, best bet is to backup and re-format, you can also try btrfs fsck but it's risky, only do that after anything important is backed up.

[TheSquigglyline]

8 minutes ago, JorgeB said:

Cache filesystem is corrupt, best bet is to backup and re-format, you can also try btrfs fsck but it's risky, only do that after anything important is backed up.

Bummer. Ok. Any idea how this happened. Would this likely have occurred because the cache drive got to full? Its only about a year old SSD. No smart errors on it ever. This is just going to be a lot of work and I would like to avoid future issues. 
Follow up. Can i just set all my shares to "use cache: no" and let the mover take care of "backing up the cache" or would you recommend moving them myself. Will the mover even work because it will try and delete the content after copying it and that will fail cuz read only.

Thanks for the speedy reply!!!

[JorgeB]

2 minutes ago, TheSquigglyline said:

Any idea how this happened. Would this likely have occurred because the cache drive got to full?

Difficult to say but possible, you should avoid running out of space.

 

2 minutes ago, TheSquigglyline said:

Can i just set all my shares to "use cache: no" and let the mover take care of "backing up the cache"

It would be cache=yes, you can try, not sure if the mover works with a read only filesystem since it won't be able to delete moved files, also don't forget to disable VM and docker services first, if the mover doesn't work you can use for example rsync or midnight commander (mc on the console).

[TheSquigglyline]

20 minutes ago, JorgeB said:

Difficult to say but possible, you should avoid running out of space.

 

It would be cache=yes, you can try, not sure if the mover works with a read only filesystem since it won't be able to delete moved files, also don't forget to disable VM and docker services first, if the mover doesn't work you can use for example rsync or midnight commander (mc on the console).

Ok thank you again for the help. i will check out rsync and midnight commander. I set all the shares that were prefer to yes and initiated the mover. I will report back what happens. I also stopped my download files and plex media files from using the cache since use ram for transcoding anyway. That should stop the cache from ever filling up. As far as my docker.img file i get warnings that is is close to filling up sometimes and i have it set to 40gb i will have to look further into that. I suspect is is something with a game server i have running. 

Here's to hoping i don't have to do much if any docker reconfiguring.