Kernel Panic Last night - Now docker is 'broken'

[SeattleBandit]

had a kernel panic bring down my server and now i keep seeing this in my logs -> /var/lib/docker/network/files/local-kv.db

 

 

some docker containers start up happily but others keep giving me these types or errors and when i try and manually start them or remove a container i get a 'server error' 

 

Jun 5 13:17:51 Nighthawks rc.docker: tautulli: Error response from daemon: failed to create endpoint tautulli on network tjvnet: failed to save bridge endpoint 7f200b4 to store: failed to update bridge store for object type *bridge.bridgeEndpoint: open /var/lib/docker/network/files/local-kv.db: read-only file system
Jun 5 13:17:51 Nighthawks rc.docker: Error: failed to start containers: tautulli

 

perhaps my docker stack was screwed up due to the unclean shutdown last night?  anyone have any suggestions?

 

FIx common problems says its unable to write to docker image because its either full or corrupted - going to try increasing the docker image size ... 

Edited June 5, 2022 by SeattleBandit

[SeattleBandit]

Jun  5 13:27:07 Nighthawks root: Fix Common Problems: Error: Unable to write to Docker Image
Jun  5 13:39:01 Nighthawks kernel: BTRFS error (device loop2): unable to find ref byte nr 5920882688 parent 0 root 7  owner 0 offset 0
Jun  5 13:39:01 Nighthawks kernel: BTRFS: Transaction aborted (error -2)
Jun  5 13:39:01 Nighthawks kernel: BTRFS: error (device loop2) in __btrfs_free_extent:3092: errno=-2 No such entry
Jun  5 13:39:01 Nighthawks kernel: BTRFS: error (device loop2) in btrfs_run_delayed_refs:2144: errno=-2 No such entry
Jun  5 13:39:01 Nighthawks rc.docker: xteve: Error response from daemon: failed to update store for object type *libnetwork.endpoint: no such file or directory
Jun  5 13:39:01 Nighthawks rc.docker: Error: failed to start containers: xteve
Jun  5 13:39:02 Nighthawks rc.docker: binhex-prowlarr: Error response from daemon: failed to create endpoint binhex-prowlarr on network bridge: failed to save bridge endpoint 2ca7ca3 to store: failed to update bridge store for object type *bridge.bridgeEndpoint: open /var/lib/docker/network/files/local-kv.db: read-only file system
Jun  5 13:39:02 Nighthawks rc.docker: Error: failed to start containers: binhex-prowlarr

 

Not clear what the BTRFS error (device loop2) means - not super excited about removing the docker.img and reinstalling everything ... increasing docker.im size did not help

[jmztaylor]

First try running a btrfs scrub.  Also post diagnostics.

 

Removing docker.img is not destructive.  All your app data should be in appdata.  If you remove it, then when you start docker again, your containers will be "missing" but going to previous apps in the apps tab, they will all be right there and just have to reinstall them.  It will keep all your previous settings.

Edited June 5, 2022 by jmztaylor

[SeattleBandit]

24 minutes ago, jmztaylor said:

First try running a btrfs scrub.  Also post diagnostics.

 

Removing docker.img is not destructive.  All your app data should be in appdata.  If you remove it, then when you start docker again, your containers will be "missing" but going to previous apps in the apps tab, they will all be right there and just have to reinstall them.  It will keep all your previous settings.

 

yeah i found a reddit post saying all your configs and stuff don't get nuked - so passwords and api keys are retained - i move my old docker image and kicked things off with a new one and am reinstalling everything and thus far no issues... i am concnerned about that btrs device loop but i'm not seeing it in the logs anymore

[jmztaylor]

try running a btrfs scrub.  Also post diagnostics.

[SeattleBandit]

it blew away my custom docker network - recreating that and i'll hopefully be back fully operational

[SeattleBandit]

running btrfs scrub and pulling diags