Help Diagnosing why Docker.log keeps filling up

[CorserMoon]

A few days ago I woke up to Plex not working and the container refusing to start. Quickly learned that the docker log was full. I rebooted the system and all was fine. I monitored the log size for the next couple days and it stayed at 1%. This morning however, back to 100%. I need help trying to figure out what is causing this issue. The docker.log is full of the errors that you see in the below screenshot. I've also attached my diagnostic zip. I've since rebooted the server to get everything working again. 

 

executor-diagnostics-20211030-0853.zip

Edited October 31, 2021 by CorserMoon

[Squid]

It's most likely not your log actually filling up, but rather the image itself due to a misconfigure of the various paths?  What does Container Size show on the Docker tab?  A number of entries in the Docker FAQ here about the image filling up.

[CorserMoon]

1 hour ago, Squid said:

It's most likely not your log actually filling up, but rather the image itself due to a misconfigure of the various paths?  What does Container Size show on the Docker tab?  A number of entries in the Docker FAQ here about the image filling up.

 

I actually checked the docker img, it's sitting at 50% and was when the log filled. Here is a screenshot I grabbed:

 

And in the screenshot in the previous post, you can see that docker.log.1 is 127mb. I grabbed it off the server to check and it is in fact 127mb and full almost entirely of those "...read only file system" errors .

[trurl]

Even though it isn't full, it is corrupt. You have to recreate and reinstall your dockers.

 

https://wiki.unraid.net/Manual/Docker_Management#Re-Create_the_Docker_image_file

 

Why have you given it 50G? 20G is often more than enough. Maybe you did fill it earlier. The usual reason for filling is an application writing to a path that isn't mapped. Linux is case-sensitive.

 

 

[CorserMoon]

1 hour ago, trurl said:

Even though it isn't full, it is corrupt. You have to recreate and reinstall your dockers.

 

https://wiki.unraid.net/Manual/Docker_Management#Re-Create_the_Docker_image_file

 

Why have you given it 50G? 20G is often more than enough. Maybe you did fill it earlier. The usual reason for filling is an application writing to a path that isn't mapped. Linux is case-sensitive.

 

 

 

How do you know the img is corrupt? 

 

I've checked the paths and they are working correctly. I increased to 50gb because I was about 80% full when using the default size. I run a lot of containers and several are ~3gb each freshly installed. 

 

I think I figured out the culprit though. I translated the container IDs in the log output to the container name. It turns out that the container(s) in question were trying to write to a secondary cache pool I created which was full. This also explains the space unavailable errors I was seeing in the syslog. I cleared some space in this cache pool and will see if that does the trick. 

Edited October 30, 2021 by CorserMoon

[Squid]

2 hours ago, CorserMoon said:

How do you know the img is corrupt? 

 

Oct 30 00:03:11 Executor kernel: verify_parent_transid: 7967 callbacks suppressed
Oct 30 00:03:11 Executor kernel: BTRFS error (device loop2): parent transid verify failed on 17584734208 wanted 1640023 found 1639989

By far the easiest way for this to happen is when the image fills up.

 

Either way, the solution is the same.  Delete the image, then reinstall from Apps - Previous Apps.  5 minutes and you're done

[CorserMoon]

Thanks for the help. I followed the guide to delete and reinstall the containers and everything worked like a charm except for swag and a couple other containers that were using a custom docker network that wasn't preserved. I created the customer network via the terminal command 

 

docker network create swagnet

 

Is there a better way to create a custom docker network that will remain if docker.img is deleted?

 

 

Edited October 31, 2021 by CorserMoon

[CorserMoon]

OK, log file filled back up again over night, docker.img still at only 57% utilization. The docker.log.1 file is filled with this:

 

 

 

Attached latest diagnostic file. I deleted and rebuilt the docker.img so that shouldn't be the issue. 

 

 

executor-diagnostics-20211031-1258.zip

Edited October 31, 2021 by CorserMoon

[Squid]

Have you run a memtest lately?  Same errors within the docker.img

[CorserMoon]

9 minutes ago, Squid said:

Have you run a memtest lately?  Same errors within the docker.img

 

I'm trying to get better at investigating so forgive my questions. But how do you know that the log errors like:

 

Oct 30 00:03:11 Executor kernel: verify_parent_transid: 7967 callbacks suppressed
Oct 30 00:03:11 Executor kernel: BTRFS error (device loop2): parent transid verify failed on 17584734208 wanted 1640023 found 1639989

 

are from docker.img?

 

I also see that those errors started after the log space errors, so maybe the docker.log is filling first and then throwing those errors that look like docker.img corruption? 

 

 

In the meantime, I'll run a memtest and see if that's ok. 

[Squid]

/dev/loop2       40G   23G   18G  57% /var/lib/docker

 

 

If you keep having trouble, I'd suggest using a folder instead of an image for the docker (Settings - Docker and switch to folder)

[CorserMoon]

1 hour ago, Squid said:

/dev/loop2       40G   23G   18G  57% /var/lib/docker

 

 

If you keep having trouble, I'd suggest using a folder instead of an image for the docker (Settings - Docker and switch to folder)

 

Thanks for the tip. I'll try that after memtest finishes a pass. 45% and no errors so far. Aside from the obvious, whats the difference between having docker as a folder vs img? 

[Squid]

Usual reason for the image getting corrupted is the image filling up.  Setting it as a folder allows it to expand up to the entire size of the drive its on without actually reserving any space in advance for it.

 

From what I'm seeing in the logs though there's no obvious reason for what's going on (hence the memtest).  So, my next shot is try a folder