Docker failing to start and maybe some more

[Chris_bacon]

Ok so, docker fails to start, I tried rebooting with docker disabled let the server start then enable it because I also got a server out of memory error (which doesn't appear anymore but the diag files were made when it did, I can produce new files if needed) but to no avail. Now I just don't know what to do...

 

I've added the diagnostic files and a screenshot of what appears when the docker tab is open after failing to start

 

Thanks to anyone who can help

vault-diagnostics-20230510-1752.zip

Edited May 10, 2023 by Chris_bacon

[Chris_bacon]

So, I may have answered my own thing... I found this Docker corruption. Could anyone confirm that this is the case and I have a lot of work in front of me to recreate something that I have no backup of (yeah I know I should have that but I don't, I'll make sure I do in the future).

 

Also, weird thing I can access my plex library...

[JorgeB]

A reboot should fix it.

[itimpi]

8 hours ago, Chris_bacon said:

So, I may have answered my own thing... I found this Docker corruption. Could anyone confirm that this is the case and I have a lot of work in front of me to recreate something that I have no backup of (yeah I know I should have that but I don't, I'll make sure I do in the future).

 

Also, weird thing I can access my plex library...

 

FYI:  If you get corruption of the docker image then it normally only takes a few minutes to fix it.   The settings for all docker containers installed via the Apps tab are stored on the flash drive and those containers can be re-instated with their settings intact via Apps->Previous Apps.

[Chris_bacon]

16 minutes ago, itimpi said:

 

FYI:  If you get corruption of the docker image then it normally only takes a few minutes to fix it.   The settings for all docker containers installed via the Apps tab are stored on the flash drive and those containers can be re-instated with their settings intact via Apps->Previous Apps.

Yeah I know but I'm still gonna have to do some work inside the containers like vaultwarden 

[Chris_bacon]

2 hours ago, JorgeB said:

A reboot should fix it.

I had did that previously and did it again and it didn't work. At firt it seems like it does you see the dockers but if I go in a different tab and come back I get the same message as the screenshot and I got the memory error message in the fix common problems plugin. Could it be that I'm running out of ram to boot as many docker as I have? 

 

Attached new diag files just in case

vault-diagnostics-20230511-0625.zip

[JorgeB]

1 minute ago, Chris_bacon said:

Could it be that I'm running out of ram to boot as many docker as I have? 

Possibly, fix that and re-test.

[Chris_bacon]

ok, so it seems to work... I managed to delete a crap ton of stuff from the appdata share (old docker that were not even actually installed). then I restarted the server with docker off and then I started it and it seemed to work so I performed a scrub and in an other tab I was looking at my ram usage, it was way down from before so I removed some dockers that I was keeping stopped but never used and I seem to have my things working reliably now.

 

I will definitely be upgrading my ram soon and set a daily backup to an external device as well as on the system using ca backup appdata...

 

thanks for the help 

[Ryonez]

I've been experiencing what appears to be this issue.

I've been shifting almost all of my docker management over to Portainer. My best guess so far is that the templates added fields back into templates that started screwing with the docker image and broke things somehow. I've been trying pretty much everything, deleting the docker img, even replaced the cache drives, reformatting the os drive. The only things kept that entire time was the config files.

There's still some dockers to shift over, but most are on Portainer, and things seem to be steady.
This issue first appeared when updating some images. Otherwise, I'd been running what I had for over a year.
 

[Chris_bacon]

ok, so I tought I was out of the woods (I might look at portainer later) but, I'm not...

in the docker tab of the gui I get this: Warning: mkdir(): Input/output error in /usr/local/emhttp/plugins/dynamix.docker.manager/include/DockerClient.php on line 351
I have access to some of the files on some shares but the one that has most of my stuff there's nothing showing (even if the data is actually there). If I sttop the array and start it again then I have access to the files, I've not started dockers yet but if it does the same as last time it'll work for a few hours then go away (I'll start dockers again in a few minutes and see). 

edit: the cache drive says unmountable... (are we getting somwhere there?)

vault-diagnostics-20230517-1004.zip

Edited May 17, 2023 by Chris_bacon

[JorgeB]

Log e completely spammed with cache device errors:

 

May 15 04:40:07 Vault kernel: sd 6:0:0:0: [sdf] tag#25 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=DRIVER_OK cmd_age=0s
May 15 04:40:07 Vault kernel: sd 6:0:0:0: [sdf] tag#25 CDB: opcode=0x28 28 00 15 68 ed 80 00 00 08 00
May 15 04:40:07 Vault kernel: blk_print_req_error: 54 callbacks suppressed
May 15 04:40:07 Vault kernel: I/O error, dev sdf, sector 359198080 op 0x0:(READ) flags 0x1000 phys_seg 1 prio class 0
May 15 04:40:07 Vault kernel: btrfs_dev_stat_print_on_error: 54 callbacks suppressed
May 15 04:40:07 Vault kernel: BTRFS error (device sdf1: state EA): bdev /dev/sdf1 errs: wr 31, rd 146725, flush 0, corrupt 0, gen 0
May 15 04:40:07 Vault kernel: sd 6:0:0:0: [sdf] tag#26 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=DRIVER_OK cmd_age=0s
May 15 04:40:07 Vault kernel: sd 6:0:0:0: [sdf] tag#26 CDB: opcode=0x28 28 00 15 68 ed 90 00 00 08 00
May 15 04:40:07 Vault kernel: I/O error, dev sdf, sector 359198096 op 0x0:(READ) flags 0x1000 phys_seg 1 prio class 0

 

Check/replace cables and post new diags after array start.

 

 

[Chris_bacon]

Cable could have gone bad? It hasn't been moved or unplugged for months... Anyway, I'll do that in the morning. Could it be the drive going bad?

 

P.s.: @JorgeBI see you on many post like mine helping, I hope it gets back to you