June 5, 20224 yr had a kernel panic bring down my server and now i keep seeing this in my logs -> /var/lib/docker/network/files/local-kv.db some docker containers start up happily but others keep giving me these types or errors and when i try and manually start them or remove a container i get a 'server error' Jun 5 13:17:51 Nighthawks rc.docker: tautulli: Error response from daemon: failed to create endpoint tautulli on network tjvnet: failed to save bridge endpoint 7f200b4 to store: failed to update bridge store for object type *bridge.bridgeEndpoint: open /var/lib/docker/network/files/local-kv.db: read-only file system Jun 5 13:17:51 Nighthawks rc.docker: Error: failed to start containers: tautulli perhaps my docker stack was screwed up due to the unclean shutdown last night? anyone have any suggestions? FIx common problems says its unable to write to docker image because its either full or corrupted - going to try increasing the docker image size ... Edited June 5, 20224 yr by SeattleBandit
June 5, 20224 yr Author Jun 5 13:27:07 Nighthawks root: Fix Common Problems: Error: Unable to write to Docker Image Jun 5 13:39:01 Nighthawks kernel: BTRFS error (device loop2): unable to find ref byte nr 5920882688 parent 0 root 7 owner 0 offset 0 Jun 5 13:39:01 Nighthawks kernel: BTRFS: Transaction aborted (error -2) Jun 5 13:39:01 Nighthawks kernel: BTRFS: error (device loop2) in __btrfs_free_extent:3092: errno=-2 No such entry Jun 5 13:39:01 Nighthawks kernel: BTRFS: error (device loop2) in btrfs_run_delayed_refs:2144: errno=-2 No such entry Jun 5 13:39:01 Nighthawks rc.docker: xteve: Error response from daemon: failed to update store for object type *libnetwork.endpoint: no such file or directory Jun 5 13:39:01 Nighthawks rc.docker: Error: failed to start containers: xteve Jun 5 13:39:02 Nighthawks rc.docker: binhex-prowlarr: Error response from daemon: failed to create endpoint binhex-prowlarr on network bridge: failed to save bridge endpoint 2ca7ca3 to store: failed to update bridge store for object type *bridge.bridgeEndpoint: open /var/lib/docker/network/files/local-kv.db: read-only file system Jun 5 13:39:02 Nighthawks rc.docker: Error: failed to start containers: binhex-prowlarr Not clear what the BTRFS error (device loop2) means - not super excited about removing the docker.img and reinstalling everything ... increasing docker.im size did not help
June 5, 20224 yr First try running a btrfs scrub. Also post diagnostics. Removing docker.img is not destructive. All your app data should be in appdata. If you remove it, then when you start docker again, your containers will be "missing" but going to previous apps in the apps tab, they will all be right there and just have to reinstall them. It will keep all your previous settings. Edited June 5, 20224 yr by jmztaylor
June 5, 20224 yr Author 24 minutes ago, jmztaylor said: First try running a btrfs scrub. Also post diagnostics. Removing docker.img is not destructive. All your app data should be in appdata. If you remove it, then when you start docker again, your containers will be "missing" but going to previous apps in the apps tab, they will all be right there and just have to reinstall them. It will keep all your previous settings. yeah i found a reddit post saying all your configs and stuff don't get nuked - so passwords and api keys are retained - i move my old docker image and kicked things off with a new one and am reinstalling everything and thus far no issues... i am concnerned about that btrs device loop but i'm not seeing it in the logs anymore
June 5, 20224 yr Author it blew away my custom docker network - recreating that and i'll hopefully be back fully operational
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.