[Solved] Docker crashed, before i remove docker.img i need some help


je82

Recommended Posts

My cache volume became full and docker crashed, i've disabled docker now and i look into the appdata folder and i see all the cached data of the applications is available there, that's good and all, but my worry is... where is all the docker configuration data stored? Like the data what the docker apps name, ip address, manually configured paths etc? Is this stored in docker.img i am essentially screwed and have to manually figure out all my settings because i never backed up my docker.img.

 

When i try to mount docker.img  or extract its data it tells me it is corrupt, i never tried this before so i don't know if it was possible or not.

 

EDIT: my docker.img is 20GB, my cache volume became full, docker never used more than 16GB, yet it became scrambled, what happened to the remaining 4gb is a question that remains

Edited by je82
Link to comment
22 minutes ago, johnnie.black said:

You can easily recreate the docker image keeping all settings, make sure to free up some space first.

Thanks for the info.

Strangely enough docker appears to work after i rebooted entire unraid. Tried to simply shutdown the docker service and start it again without rebooting unraid but it would not start.

 

For whatever reason restarting unraid solved it? What could be the cause?

 

Here's what happened: I was filling the cache to trigger the mover to transfer to array.. i forgot that on wednesdays i have a backup that runs on a server in the morning. The backup ran and i was transfering files, it ended up filling the cache completely and i even lost access via web gui for whatever reason.

 

I realized that the backup was running so i cut the network to unraid, and then i went to the console and manually went to /mnt/cache/*path where my backup file was being built*/ and then i simply rm Backup_file

 

That file cleared up 150gb, i connected the network again, the webgui came back online and the cache looked like it had space left, but docker for whatever reason was returning 403 error when i tried to start a container after stopping it.

 

I shut down the entire docker service, and attempting to start it again resulted in error.

 

Should i be worried there may be issues somewhere? I see the logserver logged a lot of IO errors during the time when smb transfers were being made but cache was full:

 

Aug 26 09:18:24 NAS kernel: BTRFS: error (device loop2) in cleanup_transaction:1846: errno=-5 IO failure

Aug 26 09:18:24 NAS kernel: BTRFS warning (device loop2): Skipping commit of aborted transaction.

Aug 26 09:18:24 NAS kernel: BTRFS info (device loop2): forced readonly

Aug 26 09:18:24 NAS kernel: BTRFS: error (device loop2) in btrfs_commit_transaction:2236: errno=-5 IO failure (Error while writing out transaction)

Aug 26 09:18:24 NAS kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 4, rd 0, flush 0, corrupt 0, gen 0

Aug 26 09:18:24 NAS kernel: print_req_error: I/O error, dev loop2, sector 645504

 

etc... but right now all systems seems to work fine..

Link to comment
10 minutes ago, johnnie.black said:

Restarting (unmounting/mounting the filesystem) might free up some space, you still need to make sure cache has enough available space now, or it will happen again.

From what i could see in the gui none of the systems reported anything being out of space after the intial fill up, the docker.img container was never above 66% usage, the cache was 100% at some point though when everything came crumbling down, just strange that things wouldn't get back to normal when i cleared up space via cli in the cache... i still had to reboot which annoys me :) i hate rebooting!

 

Since i kept getting 403 errors my guess is that BTRFS "forced readonly" was still enabled for some files on the cache drive, particularly docker.img?

Edited by je82
Link to comment
1 hour ago, johnnie.black said:

This was the problem, not the docker image, docker image stopped being able to write due to cache being full.

Yeah i hear you, but the question is, once i cleared up space on the cache, i could continue to write to it, yet docker kept returning 403 which was strange as the cache was no longer full. Then restarting the docker service wasn't helping either. Anyway i have to just move past it and make sure i don't fill the cache.

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.